Note: This is test shard 1 of 8.
[==========] Running 2 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 1 test from MaintenanceModeRF3ITest
[ RUN ] MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate
2025-01-13T08:54:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-13T08:54:09Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250113 08:54:09.109373 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.123.62:33371
--webserver_interface=127.7.123.62
--webserver_port=0
--builtin_ntp_servers=127.7.123.20:43147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.123.62:33371 with env {}
W20250113 08:54:09.496672 7668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:09.497108 7668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:09.497265 7668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:09.505803 7668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250113 08:54:09.505991 7668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:09.506074 7668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250113 08:54:09.506168 7668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 08:54:09.516744 7668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:43147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.123.62:33371
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.123.62:33371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.123.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:09.518738 7668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:09.520848 7668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:09.533676 7674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:09.534245 7673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:09.535369 7676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:09.536360 7668 server_base.cc:1034] running on GCE node
I20250113 08:54:09.537364 7668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:09.539335 7668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:09.540619 7668 hybrid_clock.cc:648] HybridClock initialized: now 1736758449540519 us; error 84 us; skew 500 ppm
I20250113 08:54:09.541046 7668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:09.548861 7668 webserver.cc:458] Webserver started at http://127.7.123.62:36111/ using document root <none> and password file <none>
I20250113 08:54:09.549885 7668 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:09.550084 7668 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:09.550470 7668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:54:09.553679 7668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data/instance:
uuid: "9fa3213b689141d9b07577d30495cd75"
format_stamp: "Formatted at 2025-01-13 08:54:09 on dist-test-slave-vxbw"
I20250113 08:54:09.554639 7668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal/instance:
uuid: "9fa3213b689141d9b07577d30495cd75"
format_stamp: "Formatted at 2025-01-13 08:54:09 on dist-test-slave-vxbw"
I20250113 08:54:09.561214 7668 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250113 08:54:09.565688 7682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:09.567637 7668 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250113 08:54:09.567890 7668 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal
uuid: "9fa3213b689141d9b07577d30495cd75"
format_stamp: "Formatted at 2025-01-13 08:54:09 on dist-test-slave-vxbw"
I20250113 08:54:09.568142 7668 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:09.593896 7668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:09.594982 7668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:09.595346 7668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:09.616142 7668 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.62:33371
I20250113 08:54:09.616161 7733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.62:33371 every 8 connection(s)
I20250113 08:54:09.617970 7668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data/info.pb
I20250113 08:54:09.622615 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 7668
I20250113 08:54:09.622953 7660 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal/instance
I20250113 08:54:09.622933 7734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:09.634671 7734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75: Bootstrap starting.
I20250113 08:54:09.638581 7734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:09.640118 7734 log.cc:826] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75: Log is configured to *not* fsync() on all Append() calls
I20250113 08:54:09.644090 7734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75: No bootstrap required, opened a new log
I20250113 08:54:09.651841 7734 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } }
I20250113 08:54:09.652453 7734 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:09.652581 7734 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa3213b689141d9b07577d30495cd75, State: Initialized, Role: FOLLOWER
I20250113 08:54:09.653403 7734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } }
I20250113 08:54:09.653649 7734 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 08:54:09.653782 7734 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 08:54:09.653982 7734 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:09.656471 7734 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } }
I20250113 08:54:09.657127 7734 leader_election.cc:304] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa3213b689141d9b07577d30495cd75; no voters:
I20250113 08:54:09.657778 7734 leader_election.cc:290] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 08:54:09.657955 7739 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:54:09.658630 7739 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 1 LEADER]: Becoming Leader. State: Replica: 9fa3213b689141d9b07577d30495cd75, State: Running, Role: LEADER
I20250113 08:54:09.659332 7739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } }
I20250113 08:54:09.660341 7734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 08:54:09.662835 7740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fa3213b689141d9b07577d30495cd75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } } }
I20250113 08:54:09.663271 7740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [sys.catalog]: This master's current role is: LEADER
I20250113 08:54:09.665110 7741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa3213b689141d9b07577d30495cd75. Latest consensus state: current_term: 1 leader_uuid: "9fa3213b689141d9b07577d30495cd75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } } }
I20250113 08:54:09.665737 7741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [sys.catalog]: This master's current role is: LEADER
I20250113 08:54:09.667236 7744 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 08:54:09.674182 7744 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 08:54:09.684871 7744 catalog_manager.cc:1348] Generated new cluster ID: 392e7551f3cc4c129852d2a40d11fb8b
I20250113 08:54:09.685119 7744 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 08:54:09.720604 7744 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 08:54:09.722273 7744 catalog_manager.cc:1505] Loading token signing keys...
I20250113 08:54:09.734329 7744 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75: Generated new TSK 0
I20250113 08:54:09.735585 7744 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 08:54:09.754436 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.1:0
--local_ip_for_outbound_sockets=127.7.123.1
--webserver_interface=127.7.123.1
--webserver_port=0
--tserver_master_addrs=127.7.123.62:33371
--builtin_ntp_servers=127.7.123.20:43147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
W20250113 08:54:10.155004 7758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:10.155397 7758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:10.155540 7758 flags.cc:425] Enabled unsafe flag: --enable_log_gc=false
W20250113 08:54:10.155623 7758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:10.164412 7758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:10.164697 7758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.1
I20250113 08:54:10.175546 7758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:43147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.123.1
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.7.123.62:33371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.1
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:10.177626 7758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:10.179984 7758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:10.194476 7763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:10.195111 7764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:10.196043 7766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:10.197070 7758 server_base.cc:1034] running on GCE node
I20250113 08:54:10.197824 7758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:10.199052 7758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:10.200285 7758 hybrid_clock.cc:648] HybridClock initialized: now 1736758450200226 us; error 61 us; skew 500 ppm
I20250113 08:54:10.200660 7758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:10.203677 7758 webserver.cc:458] Webserver started at http://127.7.123.1:36711/ using document root <none> and password file <none>
I20250113 08:54:10.204689 7758 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:10.204834 7758 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:10.205230 7758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:54:10.208655 7758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data/instance:
uuid: "aea19a9cd21144508fdcfd7b3570a0e1"
format_stamp: "Formatted at 2025-01-13 08:54:10 on dist-test-slave-vxbw"
I20250113 08:54:10.209693 7758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal/instance:
uuid: "aea19a9cd21144508fdcfd7b3570a0e1"
format_stamp: "Formatted at 2025-01-13 08:54:10 on dist-test-slave-vxbw"
I20250113 08:54:10.216470 7758 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250113 08:54:10.221024 7772 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:10.223052 7758 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250113 08:54:10.223299 7758 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal
uuid: "aea19a9cd21144508fdcfd7b3570a0e1"
format_stamp: "Formatted at 2025-01-13 08:54:10 on dist-test-slave-vxbw"
I20250113 08:54:10.223562 7758 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:10.239749 7758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:10.240890 7758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:10.241228 7758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:10.242399 7758 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:10.244304 7758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:54:10.244408 7758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:10.244539 7758 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:54:10.244627 7758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:10.280254 7758 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.1:46439
I20250113 08:54:10.280321 7884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.1:46439 every 8 connection(s)
I20250113 08:54:10.281867 7758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
I20250113 08:54:10.288995 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 7758
I20250113 08:54:10.289213 7660 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal/instance
I20250113 08:54:10.293576 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.2:0
--local_ip_for_outbound_sockets=127.7.123.2
--webserver_interface=127.7.123.2
--webserver_port=0
--tserver_master_addrs=127.7.123.62:33371
--builtin_ntp_servers=127.7.123.20:43147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
I20250113 08:54:10.299196 7885 heartbeater.cc:346] Connected to a master server at 127.7.123.62:33371
I20250113 08:54:10.299574 7885 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:10.300518 7885 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:10.303316 7699 ts_manager.cc:194] Registered new tserver with Master: aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439)
I20250113 08:54:10.305285 7699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.1:38267
W20250113 08:54:10.686578 7889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:10.687036 7889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:10.687186 7889 flags.cc:425] Enabled unsafe flag: --enable_log_gc=false
W20250113 08:54:10.687281 7889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:10.695834 7889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:10.696118 7889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.2
I20250113 08:54:10.706473 7889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:43147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.123.2
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.7.123.62:33371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.2
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:10.708494 7889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:10.710587 7889 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:10.725701 7895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:10.725795 7897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:10.726743 7894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:10.727715 7889 server_base.cc:1034] running on GCE node
I20250113 08:54:10.728425 7889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:10.729699 7889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:10.730962 7889 hybrid_clock.cc:648] HybridClock initialized: now 1736758450730878 us; error 75 us; skew 500 ppm
I20250113 08:54:10.731491 7889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:10.734241 7889 webserver.cc:458] Webserver started at http://127.7.123.2:44249/ using document root <none> and password file <none>
I20250113 08:54:10.735251 7889 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:10.735371 7889 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:10.735766 7889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:54:10.739185 7889 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/data/instance:
uuid: "1abc86bb580b4d14876623243c9d6594"
format_stamp: "Formatted at 2025-01-13 08:54:10 on dist-test-slave-vxbw"
I20250113 08:54:10.740243 7889 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/wal/instance:
uuid: "1abc86bb580b4d14876623243c9d6594"
format_stamp: "Formatted at 2025-01-13 08:54:10 on dist-test-slave-vxbw"
I20250113 08:54:10.747421 7889 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250113 08:54:10.752271 7903 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:10.753993 7889 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250113 08:54:10.754213 7889 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/wal
uuid: "1abc86bb580b4d14876623243c9d6594"
format_stamp: "Formatted at 2025-01-13 08:54:10 on dist-test-slave-vxbw"
I20250113 08:54:10.754485 7889 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:10.782676 7889 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:10.783752 7889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:10.784088 7889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:10.785190 7889 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:10.787029 7889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:54:10.787142 7889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:10.787267 7889 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:54:10.787343 7889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:10.823136 7889 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.2:45341
I20250113 08:54:10.823201 8015 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.2:45341 every 8 connection(s)
I20250113 08:54:10.824697 7889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
I20250113 08:54:10.829666 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 7889
I20250113 08:54:10.829882 7660 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-1/wal/instance
I20250113 08:54:10.834285 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.3:0
--local_ip_for_outbound_sockets=127.7.123.3
--webserver_interface=127.7.123.3
--webserver_port=0
--tserver_master_addrs=127.7.123.62:33371
--builtin_ntp_servers=127.7.123.20:43147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
I20250113 08:54:10.842020 8016 heartbeater.cc:346] Connected to a master server at 127.7.123.62:33371
I20250113 08:54:10.842446 8016 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:10.843523 8016 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:10.845574 7699 ts_manager.cc:194] Registered new tserver with Master: 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:10.846972 7699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.2:49117
W20250113 08:54:11.223019 8020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:11.223412 8020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:11.223498 8020 flags.cc:425] Enabled unsafe flag: --enable_log_gc=false
W20250113 08:54:11.223564 8020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:11.232206 8020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:11.232438 8020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.3
I20250113 08:54:11.242519 8020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:43147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.123.3
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.7.123.62:33371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.3
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:11.244626 8020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:11.246766 8020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:11.261379 8025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:11.262588 8026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:11.263337 8028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:11.264164 8020 server_base.cc:1034] running on GCE node
I20250113 08:54:11.264883 8020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:11.266217 8020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:11.267462 8020 hybrid_clock.cc:648] HybridClock initialized: now 1736758451267395 us; error 54 us; skew 500 ppm
I20250113 08:54:11.267889 8020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:11.270884 8020 webserver.cc:458] Webserver started at http://127.7.123.3:45829/ using document root <none> and password file <none>
I20250113 08:54:11.271852 8020 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:11.271992 8020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:11.272390 8020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:54:11.275935 8020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/data/instance:
uuid: "81f9c074161846b1b2a552cb12d3ef30"
format_stamp: "Formatted at 2025-01-13 08:54:11 on dist-test-slave-vxbw"
I20250113 08:54:11.276963 8020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/wal/instance:
uuid: "81f9c074161846b1b2a552cb12d3ef30"
format_stamp: "Formatted at 2025-01-13 08:54:11 on dist-test-slave-vxbw"
I20250113 08:54:11.283622 8020 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250113 08:54:11.287760 8034 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:11.289484 8020 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250113 08:54:11.289707 8020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/wal
uuid: "81f9c074161846b1b2a552cb12d3ef30"
format_stamp: "Formatted at 2025-01-13 08:54:11 on dist-test-slave-vxbw"
I20250113 08:54:11.289973 8020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:11.309216 7885 heartbeater.cc:502] Master 127.7.123.62:33371 was elected leader, sending a full tablet report...
I20250113 08:54:11.309229 8020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:11.310552 8020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:11.311003 8020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:11.312332 8020 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:11.314152 8020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:54:11.314260 8020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:11.314343 8020 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:54:11.314400 8020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:11.349541 8020 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.3:46625
I20250113 08:54:11.349586 8146 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.3:46625 every 8 connection(s)
I20250113 08:54:11.351228 8020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
I20250113 08:54:11.358345 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 8020
I20250113 08:54:11.358595 7660 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-2/wal/instance
I20250113 08:54:11.369545 8147 heartbeater.cc:346] Connected to a master server at 127.7.123.62:33371
I20250113 08:54:11.369917 8147 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:11.371006 8147 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:11.372961 7699 ts_manager.cc:194] Registered new tserver with Master: 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.373906 7699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.3:57899
I20250113 08:54:11.377072 7660 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250113 08:54:11.399683 7660 test_util.cc:274] Using random seed: -1769051176
I20250113 08:54:11.425860 7699 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37428:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000UUU\025\004\001\000\252\252\252*\004\001\000\377\377\377?\004\001\000TUUU\004\001\000\251\252\252j"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 08:54:11.428294 7699 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 08:54:11.467681 7820 tablet_service.cc:1467] Processing CreateTablet for tablet 34cc82c767a44ffab6786f89b8d41a00 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION VALUES < 357913941
I20250113 08:54:11.468698 7950 tablet_service.cc:1467] Processing CreateTablet for tablet c35e4ecbe2cc424089eaa5e0249df483 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882
I20250113 08:54:11.470085 7820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34cc82c767a44ffab6786f89b8d41a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.470883 7950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35e4ecbe2cc424089eaa5e0249df483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.471195 7818 tablet_service.cc:1467] Processing CreateTablet for tablet fe2a45627bfe4db0b9cdccfab81e2805 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823
I20250113 08:54:11.471792 7818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe2a45627bfe4db0b9cdccfab81e2805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.473145 8080 tablet_service.cc:1467] Processing CreateTablet for tablet fe2a45627bfe4db0b9cdccfab81e2805 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823
I20250113 08:54:11.474308 7947 tablet_service.cc:1467] Processing CreateTablet for tablet 44c2ecb48ca14049a051ee94f7d58cd6 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705
I20250113 08:54:11.475045 7947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c2ecb48ca14049a051ee94f7d58cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.475101 7817 tablet_service.cc:1467] Processing CreateTablet for tablet c1e64a047ae649ac9afc1b77608c067f (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764
I20250113 08:54:11.475314 8078 tablet_service.cc:1467] Processing CreateTablet for tablet 44c2ecb48ca14049a051ee94f7d58cd6 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705
I20250113 08:54:11.475625 7817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e64a047ae649ac9afc1b77608c067f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.476222 7949 tablet_service.cc:1467] Processing CreateTablet for tablet fe2a45627bfe4db0b9cdccfab81e2805 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823
I20250113 08:54:11.476454 8078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c2ecb48ca14049a051ee94f7d58cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.476577 7816 tablet_service.cc:1467] Processing CreateTablet for tablet 44c2ecb48ca14049a051ee94f7d58cd6 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705
I20250113 08:54:11.476758 7949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe2a45627bfe4db0b9cdccfab81e2805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.477058 7816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c2ecb48ca14049a051ee94f7d58cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.477738 7815 tablet_service.cc:1467] Processing CreateTablet for tablet 993904453c594e2db077d13f62d9b27a (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 1789569705 <= VALUES
I20250113 08:54:11.478227 7815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993904453c594e2db077d13f62d9b27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.478790 7819 tablet_service.cc:1467] Processing CreateTablet for tablet c35e4ecbe2cc424089eaa5e0249df483 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882
I20250113 08:54:11.479277 7819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35e4ecbe2cc424089eaa5e0249df483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.479149 8081 tablet_service.cc:1467] Processing CreateTablet for tablet c35e4ecbe2cc424089eaa5e0249df483 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882
I20250113 08:54:11.479691 8081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35e4ecbe2cc424089eaa5e0249df483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.479738 7948 tablet_service.cc:1467] Processing CreateTablet for tablet c1e64a047ae649ac9afc1b77608c067f (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764
I20250113 08:54:11.480247 7948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e64a047ae649ac9afc1b77608c067f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.473939 8079 tablet_service.cc:1467] Processing CreateTablet for tablet c1e64a047ae649ac9afc1b77608c067f (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764
I20250113 08:54:11.483384 8079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e64a047ae649ac9afc1b77608c067f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.485205 7951 tablet_service.cc:1467] Processing CreateTablet for tablet 34cc82c767a44ffab6786f89b8d41a00 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION VALUES < 357913941
I20250113 08:54:11.485936 7951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34cc82c767a44ffab6786f89b8d41a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.486900 8077 tablet_service.cc:1467] Processing CreateTablet for tablet 993904453c594e2db077d13f62d9b27a (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 1789569705 <= VALUES
I20250113 08:54:11.487519 8077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993904453c594e2db077d13f62d9b27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.491997 8080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe2a45627bfe4db0b9cdccfab81e2805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.502688 8082 tablet_service.cc:1467] Processing CreateTablet for tablet 34cc82c767a44ffab6786f89b8d41a00 (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION VALUES < 357913941
I20250113 08:54:11.503399 8082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34cc82c767a44ffab6786f89b8d41a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.468632 7946 tablet_service.cc:1467] Processing CreateTablet for tablet 993904453c594e2db077d13f62d9b27a (DEFAULT_TABLE table=test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]), partition=RANGE (key) PARTITION 1789569705 <= VALUES
I20250113 08:54:11.507514 7946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993904453c594e2db077d13f62d9b27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:11.535173 8172 tablet_bootstrap.cc:492] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
I20250113 08:54:11.540019 8173 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30: Bootstrap starting.
I20250113 08:54:11.540305 8172 tablet_bootstrap.cc:654] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.542111 8172 log.cc:826] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Log is configured to *not* fsync() on all Append() calls
I20250113 08:54:11.544113 8171 tablet_bootstrap.cc:492] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594: Bootstrap starting.
I20250113 08:54:11.546301 8173 tablet_bootstrap.cc:654] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.547225 8172 tablet_bootstrap.cc:492] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: No bootstrap required, opened a new log
I20250113 08:54:11.547606 8172 ts_tablet_manager.cc:1397] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20250113 08:54:11.547976 8171 tablet_bootstrap.cc:654] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.548035 8173 log.cc:826] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30: Log is configured to *not* fsync() on all Append() calls
I20250113 08:54:11.549381 8171 log.cc:826] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594: Log is configured to *not* fsync() on all Append() calls
I20250113 08:54:11.551826 8173 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30: No bootstrap required, opened a new log
I20250113 08:54:11.552161 8171 tablet_bootstrap.cc:492] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594: No bootstrap required, opened a new log
I20250113 08:54:11.552199 8173 ts_tablet_manager.cc:1397] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20250113 08:54:11.552451 8171 ts_tablet_manager.cc:1397] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20250113 08:54:11.557631 8172 raft_consensus.cc:357] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.558267 8172 raft_consensus.cc:383] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.558442 8172 raft_consensus.cc:738] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.559471 8172 consensus_queue.cc:260] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.559419 8171 raft_consensus.cc:357] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.559870 8171 raft_consensus.cc:383] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.560029 8171 raft_consensus.cc:738] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc86bb580b4d14876623243c9d6594, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.560933 8171 consensus_queue.cc:260] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.561939 8172 ts_tablet_manager.cc:1428] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20250113 08:54:11.562461 8172 tablet_bootstrap.cc:492] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
I20250113 08:54:11.562409 8173 raft_consensus.cc:357] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.563117 8173 raft_consensus.cc:383] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.563284 8173 raft_consensus.cc:738] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f9c074161846b1b2a552cb12d3ef30, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.564242 8173 consensus_queue.cc:260] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.564626 8172 tablet_bootstrap.cc:654] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.572901 8016 heartbeater.cc:502] Master 127.7.123.62:33371 was elected leader, sending a full tablet report...
I20250113 08:54:11.573561 8172 tablet_bootstrap.cc:492] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: No bootstrap required, opened a new log
I20250113 08:54:11.573585 8173 ts_tablet_manager.cc:1428] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30: Time spent starting tablet: real 0.021s user 0.007s sys 0.011s
I20250113 08:54:11.573702 8171 ts_tablet_manager.cc:1428] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594: Time spent starting tablet: real 0.021s user 0.015s sys 0.005s
I20250113 08:54:11.573742 8172 ts_tablet_manager.cc:1397] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250113 08:54:11.574028 8147 heartbeater.cc:502] Master 127.7.123.62:33371 was elected leader, sending a full tablet report...
I20250113 08:54:11.574246 8171 tablet_bootstrap.cc:492] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594: Bootstrap starting.
I20250113 08:54:11.575058 8172 raft_consensus.cc:357] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:11.575378 8172 raft_consensus.cc:383] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.575524 8172 raft_consensus.cc:738] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.575794 8173 tablet_bootstrap.cc:492] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30: Bootstrap starting.
I20250113 08:54:11.576026 8172 consensus_queue.cc:260] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
W20250113 08:54:11.577085 8017 tablet.cc:2367] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:54:11.577414 8172 ts_tablet_manager.cc:1428] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250113 08:54:11.577824 8172 tablet_bootstrap.cc:492] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
I20250113 08:54:11.579294 8173 tablet_bootstrap.cc:654] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.579294 8171 tablet_bootstrap.cc:654] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.580235 8172 tablet_bootstrap.cc:654] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.589041 8172 tablet_bootstrap.cc:492] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1: No bootstrap required, opened a new log
I20250113 08:54:11.589366 8172 ts_tablet_manager.cc:1397] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20250113 08:54:11.590355 8172 raft_consensus.cc:357] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.590705 8172 raft_consensus.cc:383] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.590867 8172 raft_consensus.cc:738] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.591333 8172 consensus_queue.cc:260] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.592839 8172 ts_tablet_manager.cc:1428] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 08:54:11.593258 8172 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
I20250113 08:54:11.593842 8179 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.594161 8179 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.595769 8172 tablet_bootstrap.cc:654] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.599229 8179 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
W20250113 08:54:11.603544 8148 tablet.cc:2367] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:54:11.605026 8177 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.605353 8177 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:11.607878 8177 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:11.628659 8173 tablet_bootstrap.cc:492] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30: No bootstrap required, opened a new log
I20250113 08:54:11.628924 8173 ts_tablet_manager.cc:1397] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30: Time spent bootstrapping tablet: real 0.053s user 0.027s sys 0.011s
I20250113 08:54:11.630487 8173 raft_consensus.cc:357] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:11.630873 8173 raft_consensus.cc:383] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.630971 8173 raft_consensus.cc:738] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f9c074161846b1b2a552cb12d3ef30, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.631292 8173 consensus_queue.cc:260] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:11.633620 8171 tablet_bootstrap.cc:492] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594: No bootstrap required, opened a new log
I20250113 08:54:11.633828 8171 ts_tablet_manager.cc:1397] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594: Time spent bootstrapping tablet: real 0.060s user 0.018s sys 0.033s
I20250113 08:54:11.634999 8171 raft_consensus.cc:357] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.635263 8171 raft_consensus.cc:383] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.635346 8171 raft_consensus.cc:738] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc86bb580b4d14876623243c9d6594, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.635671 8171 consensus_queue.cc:260] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.637904 8173 ts_tablet_manager.cc:1428] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250113 08:54:11.639340 8171 ts_tablet_manager.cc:1428] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250113 08:54:11.639780 8173 tablet_bootstrap.cc:492] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30: Bootstrap starting.
I20250113 08:54:11.641793 8171 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594: Bootstrap starting.
I20250113 08:54:11.642786 8173 tablet_bootstrap.cc:654] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.644076 8171 tablet_bootstrap.cc:654] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.655380 8177 raft_consensus.cc:491] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.655678 8177 raft_consensus.cc:513] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.656502 8177 leader_election.cc:290] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.657397 8171 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594: No bootstrap required, opened a new log
I20250113 08:54:11.657428 8173 tablet_bootstrap.cc:492] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30: No bootstrap required, opened a new log
I20250113 08:54:11.657618 8173 ts_tablet_manager.cc:1397] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20250113 08:54:11.657631 8171 ts_tablet_manager.cc:1397] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.003s
I20250113 08:54:11.657862 8178 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.658218 8178 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.658622 8173 raft_consensus.cc:357] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.658628 8171 raft_consensus.cc:357] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.658905 8171 raft_consensus.cc:383] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.658900 8173 raft_consensus.cc:383] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.658991 8173 raft_consensus.cc:738] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f9c074161846b1b2a552cb12d3ef30, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.659003 8171 raft_consensus.cc:738] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc86bb580b4d14876623243c9d6594, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.659284 8173 consensus_queue.cc:260] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.659300 8171 consensus_queue.cc:260] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.660010 8173 ts_tablet_manager.cc:1428] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 08:54:11.660432 8173 tablet_bootstrap.cc:492] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30: Bootstrap starting.
I20250113 08:54:11.663403 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:11.661239 8178 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.665544 8173 tablet_bootstrap.cc:654] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.665560 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" is_pre_election: true
W20250113 08:54:11.669275 8037 leader_election.cc:343] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.671912 8177 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.672152 8177 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.673034 8177 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.676084 8172 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: No bootstrap required, opened a new log
I20250113 08:54:11.676473 8172 ts_tablet_manager.cc:1397] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.001s
I20250113 08:54:11.677860 8172 raft_consensus.cc:357] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.678375 8172 raft_consensus.cc:383] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.678562 8172 raft_consensus.cc:738] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.679090 8172 consensus_queue.cc:260] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.682005 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:11.683249 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:11.684131 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:11.684492 7971 raft_consensus.cc:2463] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea19a9cd21144508fdcfd7b3570a0e1 in term 0.
W20250113 08:54:11.685506 7776 leader_election.cc:343] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:54:11.686213 7776 leader_election.cc:343] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.686574 7776 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1; no voters:
I20250113 08:54:11.688138 8177 raft_consensus.cc:2798] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:54:11.688457 8177 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:54:11.688642 8177 raft_consensus.cc:3054] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.693396 8179 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.693670 8179 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.694729 8179 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:11.692840 8177 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.699164 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" is_pre_election: true
W20250113 08:54:11.700528 8037 leader_election.cc:343] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.701357 8172 ts_tablet_manager.cc:1428] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20250113 08:54:11.702394 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594"
I20250113 08:54:11.702741 7971 raft_consensus.cc:3054] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.706315 7971 raft_consensus.cc:2463] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea19a9cd21144508fdcfd7b3570a0e1 in term 1.
I20250113 08:54:11.707377 8178 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.707623 8178 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.707883 8177 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 election: Requested vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.708324 7776 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1; no voters:
I20250113 08:54:11.708400 8178 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.707377 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:11.708761 7971 raft_consensus.cc:2463] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f9c074161846b1b2a552cb12d3ef30 in term 0.
W20250113 08:54:11.709756 8038 leader_election.cc:343] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.709961 8038 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81f9c074161846b1b2a552cb12d3ef30; no voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:11.710520 8179 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:54:11.710956 8179 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.711150 8179 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:11.711956 8179 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:11.712507 8177 raft_consensus.cc:2798] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:54:11.715217 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:11.715338 8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:11.715950 8102 raft_consensus.cc:2463] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea19a9cd21144508fdcfd7b3570a0e1 in term 0.
I20250113 08:54:11.716976 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
W20250113 08:54:11.718271 7774 leader_election.cc:343] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.718335 8038 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30; no voters: aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:11.718744 8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30"
I20250113 08:54:11.719394 8178 raft_consensus.cc:491] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.719555 7774 leader_election.cc:304] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:11.719595 8178 raft_consensus.cc:513] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.720273 8179 raft_consensus.cc:2798] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:54:11.720400 8179 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:54:11.720430 8178 leader_election.cc:290] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.720543 8179 raft_consensus.cc:3054] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.724259 8179 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.724517 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" is_pre_election: true
I20250113 08:54:11.724310 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" is_pre_election: true
I20250113 08:54:11.725569 8179 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 election: Requested vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:11.725404 7838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35e4ecbe2cc424089eaa5e0249df483" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" is_pre_election: true
I20250113 08:54:11.725818 7838 raft_consensus.cc:2463] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1abc86bb580b4d14876623243c9d6594 in term 0.
I20250113 08:54:11.726328 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
W20250113 08:54:11.726863 7774 leader_election.cc:343] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.726737 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594"
W20250113 08:54:11.727041 8038 leader_election.cc:343] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.727144 7970 raft_consensus.cc:3054] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.727370 7774 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594
I20250113 08:54:11.728093 8198 raft_consensus.cc:2743] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:54:11.728341 8172 tablet_bootstrap.cc:492] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
I20250113 08:54:11.728492 8198 raft_consensus.cc:2798] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:54:11.728681 8198 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:54:11.728839 8198 raft_consensus.cc:3054] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.730572 7838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" is_pre_election: true
I20250113 08:54:11.731375 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1"
W20250113 08:54:11.732101 8037 leader_election.cc:343] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:54:11.732157 7906 leader_election.cc:343] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.732317 8198 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:11.732887 7838 raft_consensus.cc:2388] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 81f9c074161846b1b2a552cb12d3ef30 in current term 1: Already voted for candidate aea19a9cd21144508fdcfd7b3570a0e1 in this term.
I20250113 08:54:11.732728 7906 leader_election.cc:304] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1; no voters:
I20250113 08:54:11.733417 8191 raft_consensus.cc:2798] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:54:11.733558 8191 raft_consensus.cc:491] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:54:11.733708 8191 raft_consensus.cc:3054] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.734182 7970 raft_consensus.cc:2463] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f9c074161846b1b2a552cb12d3ef30 in term 1.
W20250113 08:54:11.735162 7774 leader_election.cc:343] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.735580 8171 ts_tablet_manager.cc:1428] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594: Time spent starting tablet: real 0.078s user 0.021s sys 0.034s
I20250113 08:54:11.735558 8038 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30; no voters: aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:11.736316 8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30"
I20250113 08:54:11.736629 8102 raft_consensus.cc:3054] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.736811 8171 tablet_bootstrap.cc:492] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594: Bootstrap starting.
I20250113 08:54:11.737918 8179 raft_consensus.cc:2798] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:54:11.738246 8191 raft_consensus.cc:513] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.740070 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594"
I20250113 08:54:11.740252 8171 tablet_bootstrap.cc:654] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594: Neither blocks nor log segments found. Creating new log.
W20250113 08:54:11.741307 7776 leader_election.cc:343] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.743796 7838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35e4ecbe2cc424089eaa5e0249df483" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1"
I20250113 08:54:11.744094 7838 raft_consensus.cc:3054] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.745500 8102 raft_consensus.cc:2463] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea19a9cd21144508fdcfd7b3570a0e1 in term 1.
I20250113 08:54:11.745941 8037 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81f9c074161846b1b2a552cb12d3ef30; no voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:11.747524 7838 raft_consensus.cc:2463] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1abc86bb580b4d14876623243c9d6594 in term 1.
I20250113 08:54:11.748406 7906 leader_election.cc:304] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1; no voters:
I20250113 08:54:11.748894 8178 raft_consensus.cc:2798] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:54:11.749402 8171 tablet_bootstrap.cc:492] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594: No bootstrap required, opened a new log
I20250113 08:54:11.749680 8171 ts_tablet_manager.cc:1397] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20250113 08:54:11.751116 8171 raft_consensus.cc:357] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:11.751529 8171 raft_consensus.cc:383] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.751679 8171 raft_consensus.cc:738] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc86bb580b4d14876623243c9d6594, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.752151 8171 consensus_queue.cc:260] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:11.753784 8191 leader_election.cc:290] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 election: Requested vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.756857 8198 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 election: Requested vote from peers 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:11.759851 8199 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:54:11.760309 8179 raft_consensus.cc:695] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 1 LEADER]: Becoming Leader. State: Replica: 81f9c074161846b1b2a552cb12d3ef30, State: Running, Role: LEADER
I20250113 08:54:11.761309 8171 ts_tablet_manager.cc:1428] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20250113 08:54:11.761921 8171 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594: Bootstrap starting.
I20250113 08:54:11.763432 7774 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594
I20250113 08:54:11.764256 8198 raft_consensus.cc:2798] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:54:11.764560 8198 raft_consensus.cc:695] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 LEADER]: Becoming Leader. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Running, Role: LEADER
I20250113 08:54:11.764950 8172 tablet_bootstrap.cc:654] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.765537 8171 tablet_bootstrap.cc:654] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.765429 8198 consensus_queue.cc:237] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:11.766209 8177 raft_consensus.cc:695] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 LEADER]: Becoming Leader. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Running, Role: LEADER
I20250113 08:54:11.766582 8177 consensus_queue.cc:237] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.773142 8171 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594: No bootstrap required, opened a new log
I20250113 08:54:11.773485 8171 ts_tablet_manager.cc:1397] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20250113 08:54:11.774984 8171 raft_consensus.cc:357] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.761421 8179 consensus_queue.cc:237] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.775411 8171 raft_consensus.cc:383] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.775503 8171 raft_consensus.cc:738] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc86bb580b4d14876623243c9d6594, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.775952 8171 consensus_queue.cc:260] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.776784 8171 ts_tablet_manager.cc:1428] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250113 08:54:11.775020 8178 raft_consensus.cc:695] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 1 LEADER]: Becoming Leader. State: Replica: 1abc86bb580b4d14876623243c9d6594, State: Running, Role: LEADER
I20250113 08:54:11.778029 8178 consensus_queue.cc:237] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.780167 8171 tablet_bootstrap.cc:492] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594: Bootstrap starting.
I20250113 08:54:11.781354 8173 tablet_bootstrap.cc:492] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30: No bootstrap required, opened a new log
I20250113 08:54:11.781580 8173 ts_tablet_manager.cc:1397] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30: Time spent bootstrapping tablet: real 0.121s user 0.048s sys 0.064s
I20250113 08:54:11.782793 8171 tablet_bootstrap.cc:654] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.783083 8173 raft_consensus.cc:357] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.783413 8173 raft_consensus.cc:383] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.783515 8173 raft_consensus.cc:738] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f9c074161846b1b2a552cb12d3ef30, State: Initialized, Role: FOLLOWER
W20250113 08:54:11.785041 7886 tablet.cc:2367] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:54:11.789139 8172 tablet_bootstrap.cc:492] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1: No bootstrap required, opened a new log
I20250113 08:54:11.789603 8172 ts_tablet_manager.cc:1397] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.001s
I20250113 08:54:11.791052 8172 raft_consensus.cc:357] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.791637 8172 raft_consensus.cc:383] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.791803 8172 raft_consensus.cc:738] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.787730 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:11.786789 8173 consensus_queue.cc:260] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.792718 8172 consensus_queue.cc:260] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.787899 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35e4ecbe2cc424089eaa5e0249df483" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:11.788077 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35e4ecbe2cc424089eaa5e0249df483" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30"
I20250113 08:54:11.794945 7905 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594; no voters: 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:11.795465 8173 ts_tablet_manager.cc:1428] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20250113 08:54:11.795749 8172 ts_tablet_manager.cc:1428] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20250113 08:54:11.795768 7905 leader_election.cc:343] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.795938 8173 tablet_bootstrap.cc:492] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30: Bootstrap starting.
I20250113 08:54:11.789728 8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:11.796185 8172 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
I20250113 08:54:11.798286 8173 tablet_bootstrap.cc:654] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.798283 7696 catalog_manager.cc:5526] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1). New cstate: current_term: 1 leader_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: UNKNOWN } } }
W20250113 08:54:11.801725 7905 leader_election.cc:343] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:54:11.802213 7905 leader_election.cc:343] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:54:11.802373 7905 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594; no voters: 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:11.802634 8178 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:54:11.803177 8178 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:54:11.805262 8172 tablet_bootstrap.cc:654] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.808306 8171 tablet_bootstrap.cc:492] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594: No bootstrap required, opened a new log
I20250113 08:54:11.808537 8171 ts_tablet_manager.cc:1397] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.015s
I20250113 08:54:11.808216 7699 catalog_manager.cc:5526] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3). New cstate: current_term: 1 leader_uuid: "81f9c074161846b1b2a552cb12d3ef30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: HEALTHY } } }
I20250113 08:54:11.809417 8171 raft_consensus.cc:357] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.809657 8171 raft_consensus.cc:383] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.809764 8171 raft_consensus.cc:738] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc86bb580b4d14876623243c9d6594, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.810038 8171 consensus_queue.cc:260] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.810598 8171 ts_tablet_manager.cc:1428] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 08:54:11.812132 8172 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: No bootstrap required, opened a new log
I20250113 08:54:11.812420 8172 ts_tablet_manager.cc:1397] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20250113 08:54:11.812565 8173 tablet_bootstrap.cc:492] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30: No bootstrap required, opened a new log
I20250113 08:54:11.812783 8173 ts_tablet_manager.cc:1397] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20250113 08:54:11.813434 8172 raft_consensus.cc:357] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.813709 8172 raft_consensus.cc:383] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.813577 8173 raft_consensus.cc:357] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.813808 8172 raft_consensus.cc:738] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.813831 8173 raft_consensus.cc:383] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.813908 8173 raft_consensus.cc:738] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f9c074161846b1b2a552cb12d3ef30, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.814113 8172 consensus_queue.cc:260] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.814172 8173 consensus_queue.cc:260] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.814706 8173 ts_tablet_manager.cc:1428] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 08:54:11.814843 8172 ts_tablet_manager.cc:1428] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 08:54:11.815093 8173 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30: Bootstrap starting.
I20250113 08:54:11.814607 7697 catalog_manager.cc:5526] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1abc86bb580b4d14876623243c9d6594 (127.7.123.2). New cstate: current_term: 1 leader_uuid: "1abc86bb580b4d14876623243c9d6594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: UNKNOWN } } }
I20250113 08:54:11.816700 8173 tablet_bootstrap.cc:654] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:11.822100 8173 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30: No bootstrap required, opened a new log
I20250113 08:54:11.822610 7699 catalog_manager.cc:5526] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1). New cstate: current_term: 1 leader_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20250113 08:54:11.823311 8173 ts_tablet_manager.cc:1397] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20250113 08:54:11.824386 8173 raft_consensus.cc:357] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.824659 8173 raft_consensus.cc:383] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:11.824769 8173 raft_consensus.cc:738] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f9c074161846b1b2a552cb12d3ef30, State: Initialized, Role: FOLLOWER
I20250113 08:54:11.825035 8173 consensus_queue.cc:260] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.825640 8198 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.825857 8198 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.826725 8198 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.827734 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:11.827980 8173 ts_tablet_manager.cc:1428] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250113 08:54:11.828178 7970 raft_consensus.cc:2388] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aea19a9cd21144508fdcfd7b3570a0e1 in current term 1: Already voted for candidate 81f9c074161846b1b2a552cb12d3ef30 in this term.
I20250113 08:54:11.828975 8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:11.829696 7774 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:11.830040 8198 raft_consensus.cc:3054] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.833771 8198 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:54:11.841423 8202 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.841678 8202 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:11.842396 8203 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.842430 8178 raft_consensus.cc:491] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.842625 8178 raft_consensus.cc:513] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.842577 8203 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.842885 8202 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.843433 8178 leader_election.cc:290] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.843593 8203 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.843891 8198 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.844311 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" is_pre_election: true
I20250113 08:54:11.844348 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" is_pre_election: true
I20250113 08:54:11.844285 7838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" is_pre_election: true
I20250113 08:54:11.844674 7838 raft_consensus.cc:2463] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1abc86bb580b4d14876623243c9d6594 in term 0.
I20250113 08:54:11.844094 8198 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.845124 7906 leader_election.cc:304] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1; no voters:
I20250113 08:54:11.845199 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:11.845471 8099 raft_consensus.cc:2463] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1abc86bb580b4d14876623243c9d6594 in term 0.
I20250113 08:54:11.845479 7839 raft_consensus.cc:2463] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1abc86bb580b4d14876623243c9d6594 in term 0.
I20250113 08:54:11.845706 8198 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.845857 8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:11.846104 8102 raft_consensus.cc:2388] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1abc86bb580b4d14876623243c9d6594 in current term 1: Already voted for candidate aea19a9cd21144508fdcfd7b3570a0e1 in this term.
I20250113 08:54:11.846215 8203 raft_consensus.cc:2798] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:54:11.846340 8203 raft_consensus.cc:491] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:54:11.846432 8203 raft_consensus.cc:3054] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.846719 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:11.846874 8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:11.847038 7970 raft_consensus.cc:2463] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea19a9cd21144508fdcfd7b3570a0e1 in term 0.
I20250113 08:54:11.847056 8102 raft_consensus.cc:2463] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aea19a9cd21144508fdcfd7b3570a0e1 in term 0.
I20250113 08:54:11.847543 7774 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1; no voters:
I20250113 08:54:11.847826 7906 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1; no voters:
I20250113 08:54:11.848186 8178 raft_consensus.cc:2798] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:54:11.848327 8178 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:54:11.848441 8178 raft_consensus.cc:3054] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.848469 8198 raft_consensus.cc:2798] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:54:11.848704 8198 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:54:11.848872 8198 raft_consensus.cc:3054] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.850239 8203 raft_consensus.cc:513] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.850461 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:11.850770 8100 raft_consensus.cc:2463] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1abc86bb580b4d14876623243c9d6594 in term 0.
I20250113 08:54:11.851176 7905 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594; no voters: 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:11.851251 8203 leader_election.cc:290] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 election: Requested vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.851491 8202 raft_consensus.cc:3054] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.851908 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1"
I20250113 08:54:11.852160 7839 raft_consensus.cc:3054] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.852495 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30"
I20250113 08:54:11.852746 8100 raft_consensus.cc:3054] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.853335 8198 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.854223 8198 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 election: Requested vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.855070 8202 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:54:11.855595 7839 raft_consensus.cc:2463] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1abc86bb580b4d14876623243c9d6594 in term 1.
I20250113 08:54:11.855898 8100 raft_consensus.cc:2463] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1abc86bb580b4d14876623243c9d6594 in term 1.
I20250113 08:54:11.856412 7906 leader_election.cc:304] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1; no voters:
I20250113 08:54:11.856905 8202 raft_consensus.cc:2798] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:54:11.857198 8202 raft_consensus.cc:695] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 1 LEADER]: Becoming Leader. State: Replica: 1abc86bb580b4d14876623243c9d6594, State: Running, Role: LEADER
I20250113 08:54:11.857450 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30"
I20250113 08:54:11.857501 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594"
I20250113 08:54:11.857820 8100 raft_consensus.cc:3054] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.857853 8202 consensus_queue.cc:237] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.857376 8178 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.859697 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1"
I20250113 08:54:11.860100 7839 raft_consensus.cc:2388] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1abc86bb580b4d14876623243c9d6594 in current term 1: Already voted for candidate aea19a9cd21144508fdcfd7b3570a0e1 in this term.
I20250113 08:54:11.861004 8178 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 election: Requested vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:11.861248 8100 raft_consensus.cc:2463] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea19a9cd21144508fdcfd7b3570a0e1 in term 1.
I20250113 08:54:11.862406 7774 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1; no voters:
I20250113 08:54:11.862965 8198 raft_consensus.cc:2798] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:54:11.863087 7970 raft_consensus.cc:2388] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aea19a9cd21144508fdcfd7b3570a0e1 in current term 1: Already voted for candidate 1abc86bb580b4d14876623243c9d6594 in this term.
I20250113 08:54:11.863817 8198 raft_consensus.cc:695] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 LEADER]: Becoming Leader. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Running, Role: LEADER
I20250113 08:54:11.864233 8198 consensus_queue.cc:237] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.864198 8179 consensus_queue.cc:1035] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.865831 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30"
I20250113 08:54:11.866318 8100 raft_consensus.cc:2388] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1abc86bb580b4d14876623243c9d6594 in current term 1: Already voted for candidate aea19a9cd21144508fdcfd7b3570a0e1 in this term.
I20250113 08:54:11.866880 8179 consensus_queue.cc:1035] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.867044 7905 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594; no voters: 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:11.868556 8178 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250113 08:54:11.869311 7697 catalog_manager.cc:5526] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1abc86bb580b4d14876623243c9d6594 (127.7.123.2). New cstate: current_term: 1 leader_uuid: "1abc86bb580b4d14876623243c9d6594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: UNKNOWN } } }
I20250113 08:54:11.878959 8202 consensus_queue.cc:1035] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.879184 7697 catalog_manager.cc:5526] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1). New cstate: current_term: 1 leader_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: UNKNOWN } } }
I20250113 08:54:11.879397 8198 consensus_queue.cc:1035] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.884610 8198 consensus_queue.cc:1035] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.887529 8179 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:11.887457 8198 consensus_queue.cc:1035] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.887813 8179 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:11.888562 8102 raft_consensus.cc:3054] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.892989 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:11.894232 8179 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:11.898310 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" is_pre_election: true
I20250113 08:54:11.899439 8037 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81f9c074161846b1b2a552cb12d3ef30; no voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:11.902379 8199 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 08:54:11.903468 8099 raft_consensus.cc:3054] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:11.907377 8198 consensus_queue.cc:1035] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.909442 8202 consensus_queue.cc:1035] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.945994 8191 consensus_queue.cc:1035] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.958070 8191 consensus_queue.cc:1035] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.969825 8198 consensus_queue.cc:1035] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.977712 8198 consensus_queue.cc:1035] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:11.982195 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.4:0
--local_ip_for_outbound_sockets=127.7.123.4
--webserver_interface=127.7.123.4
--webserver_port=0
--tserver_master_addrs=127.7.123.62:33371
--builtin_ntp_servers=127.7.123.20:43147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
I20250113 08:54:12.459784 8198 raft_consensus.cc:491] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1abc86bb580b4d14876623243c9d6594)
I20250113 08:54:12.460179 8198 raft_consensus.cc:513] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:12.461156 8198 leader_election.cc:290] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:12.475684 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:12.480242 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:12.483961 7776 leader_election.cc:304] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:12.484413 8268 raft_consensus.cc:2743] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:13.297907 8238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:13.298451 8238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:13.298635 8238 flags.cc:425] Enabled unsafe flag: --enable_log_gc=false
W20250113 08:54:13.298746 8238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:13.314381 8238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:13.314741 8238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.4
I20250113 08:54:13.332129 8238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:43147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.7.123.4
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.7.123.62:33371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.4
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:13.334724 8238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:13.337934 8238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:13.396566 8277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:13.403856 8278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:13.427167 8238 server_base.cc:1034] running on GCE node
W20250113 08:54:13.428920 8280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:13.430212 8238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:13.436919 8238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:13.443159 8238 hybrid_clock.cc:648] HybridClock initialized: now 1736758453441034 us; error 2118 us; skew 500 ppm
I20250113 08:54:13.447726 8238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:13.458220 8238 webserver.cc:458] Webserver started at http://127.7.123.4:46163/ using document root <none> and password file <none>
I20250113 08:54:13.459896 8238 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:13.460219 8238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:13.461099 8238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:54:13.472577 8238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/data/instance:
uuid: "7eb1620e70d84c44b17b3e0aad7c38bc"
format_stamp: "Formatted at 2025-01-13 08:54:13 on dist-test-slave-vxbw"
I20250113 08:54:13.474552 8238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/wal/instance:
uuid: "7eb1620e70d84c44b17b3e0aad7c38bc"
format_stamp: "Formatted at 2025-01-13 08:54:13 on dist-test-slave-vxbw"
I20250113 08:54:13.491042 8238 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.007s sys 0.003s
I20250113 08:54:13.506745 8286 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:13.511406 8238 fs_manager.cc:730] Time spent opening block manager: real 0.016s user 0.000s sys 0.005s
I20250113 08:54:13.511916 8238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/wal
uuid: "7eb1620e70d84c44b17b3e0aad7c38bc"
format_stamp: "Formatted at 2025-01-13 08:54:13 on dist-test-slave-vxbw"
I20250113 08:54:13.512490 8238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:13.802903 8238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:13.804349 8238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:13.804751 8238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:13.848768 8238 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:13.858650 8238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:54:13.858853 8238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:13.859005 8238 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:54:13.859083 8238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250113 08:54:14.567739 7774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.123.3:46625, user_credentials={real_user=slave}} blocked reactor thread for 55756.7us
W20250113 08:54:14.679203 7774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.123.3:46625, user_credentials={real_user=slave}} blocked reactor thread for 55527.1us
W20250113 08:54:14.680606 7776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.123.2:45341, user_credentials={real_user=slave}} blocked reactor thread for 69132.6us
W20250113 08:54:16.852842 8234 meta_cache.cc:1261] Time spent looking up entry by key: real 0.074s user 0.001s sys 0.000s
I20250113 08:54:17.123755 8238 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.4:38355
I20250113 08:54:17.123775 8405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.4:38355 every 8 connection(s)
I20250113 08:54:17.126528 8238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
I20250113 08:54:17.130015 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 8238
I20250113 08:54:17.130232 7660 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-3/wal/instance
I20250113 08:54:17.403750 8406 heartbeater.cc:346] Connected to a master server at 127.7.123.62:33371
I20250113 08:54:17.404237 8406 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:17.405673 8406 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:17.408160 7697 ts_manager.cc:194] Registered new tserver with Master: 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355)
I20250113 08:54:17.409408 7697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.4:46581
I20250113 08:54:18.181033 7697 ts_manager.cc:292] Set tserver state for aea19a9cd21144508fdcfd7b3570a0e1 to MAINTENANCE_MODE
I20250113 08:54:18.182791 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 7758
I20250113 08:54:18.184487 7880 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:54:18.416705 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 7758
I20250113 08:54:18.417876 8406 heartbeater.cc:502] Master 127.7.123.62:33371 was elected leader, sending a full tablet report...
W20250113 08:54:18.448472 7904 connection.cc:537] server connection from 127.7.123.1:45183 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:54:18.449043 8037 connection.cc:537] client connection to 127.7.123.1:46439 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:54:18.449326 7906 connection.cc:537] client connection to 127.7.123.1:46439 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:54:18.448753 8158 meta_cache.cc:302] tablet 34cc82c767a44ffab6786f89b8d41a00: replica aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439) has failed: Network error: recv got EOF from 127.7.123.1:46439 (error 108)
W20250113 08:54:18.449628 8037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:54:18.449635 7906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:54:18.450671 8037 connection.cc:537] server connection from 127.7.123.1:34957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:54:18.453071 7929 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.453536 7927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.453794 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.456264 7927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.456264 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.456516 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.458315 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:18.458509 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:18.458608 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.459470 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.459560 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.459810 8060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.461342 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.461457 8060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.461623 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.462111 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:18.466958 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.468446 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.468744 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.470938 7927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.471753 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
I20250113 08:54:18.477952 8269 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aea19a9cd21144508fdcfd7b3570a0e1)
I20250113 08:54:18.478189 8269 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:18.478966 8269 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
W20250113 08:54:18.479997 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
I20250113 08:54:18.480998 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 2 candidate_status { last_received { term: 1 index: 200 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
W20250113 08:54:18.482012 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.483160 7906 leader_election.cc:336] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
I20250113 08:54:18.483322 7906 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594; no voters: 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:18.483631 8178 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:18.483882 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.483954 7927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.485832 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.485944 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.491551 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.495256 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.495313 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.500072 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.500113 8060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.500339 8062 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.500653 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.501328 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.501631 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.502384 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.504328 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.507220 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.509030 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.511178 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.511904 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.518573 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.523427 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.531483 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.533515 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.533905 7927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.535537 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.537107 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.541836 7927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.542939 7927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.544272 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.545778 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.545941 7927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.545970 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.545953 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.548467 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.549175 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.557806 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.558878 7927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.559530 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.572001 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.572010 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.578699 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.586190 7927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.586189 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.587155 7929 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.588774 7929 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.590026 7929 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.591513 7929 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.597566 7929 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.597633 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.600033 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.603991 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.604609 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.605780 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.608527 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.616935 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.616930 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.633827 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.638828 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.638783 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.642160 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.642325 7929 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.643762 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.643790 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.649524 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.654058 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.658066 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.662103 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.665256 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.671653 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.677945 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.680059 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.692693 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.698770 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.700407 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.704332 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.704383 8059 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
I20250113 08:54:18.710041 8269 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aea19a9cd21144508fdcfd7b3570a0e1)
I20250113 08:54:18.710335 8269 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:18.711045 8269 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:18.711684 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
W20250113 08:54:18.712637 7906 leader_election.cc:336] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
I20250113 08:54:18.712770 7906 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594; no voters: 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:18.713110 8269 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:18.713377 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
I20250113 08:54:18.715119 8269 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aea19a9cd21144508fdcfd7b3570a0e1)
I20250113 08:54:18.715324 8269 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:18.716039 8269 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:18.716663 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "1abc86bb580b4d14876623243c9d6594" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
W20250113 08:54:18.717489 7906 leader_election.cc:336] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
I20250113 08:54:18.717628 7906 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594; no voters: 81f9c074161846b1b2a552cb12d3ef30, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:54:18.717926 8269 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:18.718497 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
W20250113 08:54:18.720297 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
I20250113 08:54:18.724145 8430 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aea19a9cd21144508fdcfd7b3570a0e1)
I20250113 08:54:18.724400 8430 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:18.724766 8411 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aea19a9cd21144508fdcfd7b3570a0e1)
I20250113 08:54:18.724929 8411 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
W20250113 08:54:18.725059 8061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50434: Illegal state: replica 81f9c074161846b1b2a552cb12d3ef30 is not leader of this config: current role FOLLOWER
I20250113 08:54:18.725214 8430 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:18.725690 8411 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:18.725853 8411 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aea19a9cd21144508fdcfd7b3570a0e1)
I20250113 08:54:18.725971 8411 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:18.726464 8411 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:18.726594 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:18.726910 7970 raft_consensus.cc:2463] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f9c074161846b1b2a552cb12d3ef30 in term 1.
I20250113 08:54:18.727000 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 2 candidate_status { last_received { term: 1 index: 201 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:18.727066 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:18.727263 7969 raft_consensus.cc:2463] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f9c074161846b1b2a552cb12d3ef30 in term 1.
I20250113 08:54:18.727335 7971 raft_consensus.cc:2463] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f9c074161846b1b2a552cb12d3ef30 in term 1.
I20250113 08:54:18.727649 8038 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30; no voters:
I20250113 08:54:18.727998 8411 raft_consensus.cc:2798] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:54:18.728022 8038 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30; no voters:
I20250113 08:54:18.728127 8411 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aea19a9cd21144508fdcfd7b3570a0e1)
W20250113 08:54:18.728199 8037 leader_election.cc:336] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
I20250113 08:54:18.728354 8038 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30; no voters:
I20250113 08:54:18.728250 8411 raft_consensus.cc:3054] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:54:18.728646 8270 raft_consensus.cc:2798] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250113 08:54:18.728717 8037 leader_election.cc:336] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
W20250113 08:54:18.728755 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
I20250113 08:54:18.728757 8270 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aea19a9cd21144508fdcfd7b3570a0e1)
I20250113 08:54:18.728907 8270 raft_consensus.cc:3054] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:54:18.729135 8430 raft_consensus.cc:2798] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250113 08:54:18.729275 8430 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aea19a9cd21144508fdcfd7b3570a0e1)
I20250113 08:54:18.729408 8430 raft_consensus.cc:3054] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:54:18.730734 8037 leader_election.cc:336] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
I20250113 08:54:18.732223 8270 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:18.732867 8411 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:18.733035 8430 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:18.733362 8270 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 election: Requested vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:18.733745 8430 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 election: Requested vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:18.734041 8411 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 election: Requested vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:18.734321 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594"
W20250113 08:54:18.734448 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
I20250113 08:54:18.734484 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 2 candidate_status { last_received { term: 1 index: 201 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594"
I20250113 08:54:18.734655 7971 raft_consensus.cc:3054] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:54:18.734704 7969 raft_consensus.cc:3054] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:54:18.735199 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "81f9c074161846b1b2a552cb12d3ef30" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594"
I20250113 08:54:18.735492 7970 raft_consensus.cc:3054] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:54:18.737793 7971 raft_consensus.cc:2463] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f9c074161846b1b2a552cb12d3ef30 in term 2.
I20250113 08:54:18.738031 7969 raft_consensus.cc:2463] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f9c074161846b1b2a552cb12d3ef30 in term 2.
I20250113 08:54:18.738698 7970 raft_consensus.cc:2463] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f9c074161846b1b2a552cb12d3ef30 in term 2.
I20250113 08:54:18.739030 8038 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30; no voters:
I20250113 08:54:18.739370 8411 raft_consensus.cc:2798] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:54:18.739418 8038 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30; no voters:
I20250113 08:54:18.739639 8411 raft_consensus.cc:695] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 LEADER]: Becoming Leader. State: Replica: 81f9c074161846b1b2a552cb12d3ef30, State: Running, Role: LEADER
I20250113 08:54:18.739918 8038 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30; no voters:
W20250113 08:54:18.740142 8037 leader_election.cc:336] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
I20250113 08:54:18.740474 8430 raft_consensus.cc:2798] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:54:18.740947 8430 raft_consensus.cc:695] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 LEADER]: Becoming Leader. State: Replica: 81f9c074161846b1b2a552cb12d3ef30, State: Running, Role: LEADER
W20250113 08:54:18.741186 8037 leader_election.cc:336] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
I20250113 08:54:18.741219 8430 consensus_queue.cc:237] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 201, Committed index: 201, Last appended: 1.201, Last appended by leader: 201, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
W20250113 08:54:18.741392 8037 leader_election.cc:336] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
I20250113 08:54:18.740757 8270 raft_consensus.cc:2798] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:54:18.740365 8411 consensus_queue.cc:237] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 1.202, Last appended by leader: 202, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
W20250113 08:54:18.742378 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
I20250113 08:54:18.747467 8270 raft_consensus.cc:695] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 LEADER]: Becoming Leader. State: Replica: 81f9c074161846b1b2a552cb12d3ef30, State: Running, Role: LEADER
I20250113 08:54:18.747862 8270 consensus_queue.cc:237] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 1.202, Last appended by leader: 201, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:18.749009 7698 catalog_manager.cc:5526] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 reported cstate change: term changed from 1 to 2, leader changed from aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1) to 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3). New cstate: current_term: 2 leader_uuid: "81f9c074161846b1b2a552cb12d3ef30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: HEALTHY } } }
I20250113 08:54:18.750921 7970 raft_consensus.cc:1270] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Refusing update from remote peer 81f9c074161846b1b2a552cb12d3ef30: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 2 index: 204. (index mismatch)
I20250113 08:54:18.751622 8432 consensus_queue.cc:1035] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
W20250113 08:54:18.752018 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:18.755525 7931 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37620: Illegal state: replica 1abc86bb580b4d14876623243c9d6594 is not leader of this config: current role FOLLOWER
I20250113 08:54:18.756482 7970 raft_consensus.cc:1270] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Refusing update from remote peer 81f9c074161846b1b2a552cb12d3ef30: Log matching property violated. Preceding OpId in replica: term: 1 index: 200. Preceding OpId from leader: term: 2 index: 203. (index mismatch)
W20250113 08:54:18.756949 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:54:18.757246 8249 mvcc.cc:204] Tried to move back new op lower bound from 7113762647037411328 to 7113762647008550912. Current Snapshot: MvccSnapshot[applied={T|T < 7113762647037411328 or (T in {7113762647037411328})}]
I20250113 08:54:18.757807 8411 consensus_queue.cc:1035] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 202, Last known committed idx: 196, Time since last communication: 0.000s
I20250113 08:54:18.779956 7696 catalog_manager.cc:5526] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 reported cstate change: term changed from 1 to 2, leader changed from aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1) to 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3). New cstate: current_term: 2 leader_uuid: "81f9c074161846b1b2a552cb12d3ef30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } }
W20250113 08:54:18.780441 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:54:18.780478 7970 raft_consensus.cc:1270] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Refusing update from remote peer 81f9c074161846b1b2a552cb12d3ef30: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 2 index: 204. (index mismatch)
I20250113 08:54:18.781509 8431 consensus_queue.cc:1035] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20250113 08:54:18.784921 7698 catalog_manager.cc:5526] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 reported cstate change: term changed from 1 to 2, leader changed from aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1) to 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3). New cstate: current_term: 2 leader_uuid: "81f9c074161846b1b2a552cb12d3ef30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: UNKNOWN } } }
I20250113 08:54:18.788785 8224 mvcc.cc:204] Tried to move back new op lower bound from 7113762647153192960 to 7113762647035002880. Current Snapshot: MvccSnapshot[applied={T|T < 7113762647153192960 or (T in {7113762647153192960})}]
W20250113 08:54:18.874895 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:18.937242 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:18.965083 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:19.267328 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:19.269764 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:19.300302 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:19.396471 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:19.497861 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:19.504253 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:19.779806 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:19.798604 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:19.807384 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:19.975944 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:19.981647 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:20.020308 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:20.289770 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:20.298398 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:20.349582 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250113 08:54:20.496626 8453 consensus_queue.cc:579] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.073s)
W20250113 08:54:20.505350 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250113 08:54:20.524853 8203 consensus_queue.cc:579] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.104s)
W20250113 08:54:20.528627 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250113 08:54:20.545406 8203 consensus_queue.cc:579] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.125s)
W20250113 08:54:20.554728 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250113 08:54:20.749665 8453 consensus_queue.cc:579] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.008s)
I20250113 08:54:20.793342 8411 consensus_queue.cc:579] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.052s)
W20250113 08:54:20.799710 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250113 08:54:20.803711 8270 consensus_queue.cc:579] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.055s)
W20250113 08:54:20.807905 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250113 08:54:20.894425 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250113 08:54:21.022025 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:21.082490 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:21.090827 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:21.330327 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:21.365810 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:21.420178 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:21.570379 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20250113 08:54:21.631527 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20250113 08:54:21.636965 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20250113 08:54:21.684082 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 7668
I20250113 08:54:21.685431 7729 generic_service.cc:196] Checking for leaks (request via RPC)
W20250113 08:54:21.833250 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20250113 08:54:21.868301 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20250113 08:54:21.944981 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 7668
W20250113 08:54:21.956574 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20250113 08:54:21.992384 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.123.62:33371
--webserver_interface=127.7.123.62
--webserver_port=36111
--builtin_ntp_servers=127.7.123.20:43147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.123.62:33371 with env {}
W20250113 08:54:22.135234 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250113 08:54:22.154237 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250113 08:54:22.231137 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250113 08:54:22.372153 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250113 08:54:22.439136 8406 heartbeater.cc:643] Failed to heartbeat to 127.7.123.62:33371 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.123.62:33371: connect: Connection refused (error 111)
W20250113 08:54:22.455750 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250113 08:54:22.549333 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250113 08:54:22.605269 8016 heartbeater.cc:643] Failed to heartbeat to 127.7.123.62:33371 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.123.62:33371: connect: Connection refused (error 111)
W20250113 08:54:22.621209 8462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:22.621902 8462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:22.622146 8462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:22.628278 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20250113 08:54:22.633785 8462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250113 08:54:22.633934 8462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:22.634011 8462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250113 08:54:22.634182 8462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 08:54:22.650867 8462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:43147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.123.62:33371
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.123.62:33371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.123.62
--webserver_port=36111
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:22.653857 8462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:22.657020 8462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:22.662685 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20250113 08:54:22.704793 8471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:22.704984 8470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:22.708122 8473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:22.748430 8462 server_base.cc:1034] running on GCE node
I20250113 08:54:22.750156 8462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:22.761343 8462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:22.762832 8462 hybrid_clock.cc:648] HybridClock initialized: now 1736758462762713 us; error 41 us; skew 500 ppm
I20250113 08:54:22.766731 8462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250113 08:54:22.783634 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20250113 08:54:22.785060 8462 webserver.cc:458] Webserver started at http://127.7.123.62:36111/ using document root <none> and password file <none>
I20250113 08:54:22.786442 8462 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:22.786710 8462 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:22.807034 8462 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.008s sys 0.000s
I20250113 08:54:22.819377 8479 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:22.835626 8462 fs_manager.cc:730] Time spent opening block manager: real 0.026s user 0.008s sys 0.000s
I20250113 08:54:22.836045 8462 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal
uuid: "9fa3213b689141d9b07577d30495cd75"
format_stamp: "Formatted at 2025-01-13 08:54:09 on dist-test-slave-vxbw"
I20250113 08:54:22.837544 8462 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250113 08:54:22.937469 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20250113 08:54:22.981092 8147 heartbeater.cc:643] Failed to heartbeat to 127.7.123.62:33371 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.123.62:33371: connect: Connection refused (error 111)
W20250113 08:54:23.023228 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20250113 08:54:23.074663 8462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:23.076407 8462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:23.076848 8462 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250113 08:54:23.107683 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20250113 08:54:23.148655 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20250113 08:54:23.225237 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20250113 08:54:23.362232 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20250113 08:54:23.450603 8037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111) [suppressed 183 similar messages]
I20250113 08:54:23.459038 8532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.62:33371 every 8 connection(s)
I20250113 08:54:23.459515 8462 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.62:33371
I20250113 08:54:23.462059 8462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/master-0/data/info.pb
I20250113 08:54:23.472385 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 8462
W20250113 08:54:23.474223 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20250113 08:54:23.493201 8533 sys_catalog.cc:263] Verifying existing consensus state
I20250113 08:54:23.498551 8533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75: Bootstrap starting.
W20250113 08:54:23.522879 7906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20250113 08:54:23.525135 8533 log.cc:826] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75: Log is configured to *not* fsync() on all Append() calls
W20250113 08:54:23.635584 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20250113 08:54:23.637955 8533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=11 ignored=0} mutations{seen=15 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:54:23.638792 8533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75: Bootstrap complete.
I20250113 08:54:23.655504 8533 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } }
I20250113 08:54:23.658098 8533 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa3213b689141d9b07577d30495cd75, State: Initialized, Role: FOLLOWER
I20250113 08:54:23.659160 8533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } }
I20250113 08:54:23.659554 8533 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 08:54:23.659713 8533 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 08:54:23.659950 8533 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:54:23.666136 8533 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } }
I20250113 08:54:23.667068 8533 leader_election.cc:304] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa3213b689141d9b07577d30495cd75; no voters:
W20250113 08:54:23.672973 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20250113 08:54:23.673527 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20250113 08:54:23.685211 8538 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 2 FOLLOWER]: Leader election won for term 2
I20250113 08:54:23.685561 8533 leader_election.cc:290] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250113 08:54:23.695116 8538 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [term 2 LEADER]: Becoming Leader. State: Replica: 9fa3213b689141d9b07577d30495cd75, State: Running, Role: LEADER
I20250113 08:54:23.698132 8538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } }
I20250113 08:54:23.699481 8533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 08:54:23.706679 8016 heartbeater.cc:346] Connected to a master server at 127.7.123.62:33371
I20250113 08:54:23.712291 8540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9fa3213b689141d9b07577d30495cd75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } } }
I20250113 08:54:23.712668 8540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [sys.catalog]: This master's current role is: LEADER
I20250113 08:54:23.713186 8539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa3213b689141d9b07577d30495cd75. Latest consensus state: current_term: 2 leader_uuid: "9fa3213b689141d9b07577d30495cd75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3213b689141d9b07577d30495cd75" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 33371 } } }
I20250113 08:54:23.713430 8539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75 [sys.catalog]: This master's current role is: LEADER
W20250113 08:54:23.722616 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20250113 08:54:23.763069 8545 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 08:54:23.799971 8545 catalog_manager.cc:670] Loaded metadata for table test-workload [id=7b7cc3ea430140598296ffa5c84d80d5]
I20250113 08:54:23.853559 8545 tablet_loader.cc:96] loaded metadata for tablet 34cc82c767a44ffab6786f89b8d41a00 (table test-workload [id=7b7cc3ea430140598296ffa5c84d80d5])
I20250113 08:54:23.876770 8545 tablet_loader.cc:96] loaded metadata for tablet 44c2ecb48ca14049a051ee94f7d58cd6 (table test-workload [id=7b7cc3ea430140598296ffa5c84d80d5])
I20250113 08:54:23.882266 8545 tablet_loader.cc:96] loaded metadata for tablet 993904453c594e2db077d13f62d9b27a (table test-workload [id=7b7cc3ea430140598296ffa5c84d80d5])
I20250113 08:54:23.883143 8545 tablet_loader.cc:96] loaded metadata for tablet c1e64a047ae649ac9afc1b77608c067f (table test-workload [id=7b7cc3ea430140598296ffa5c84d80d5])
I20250113 08:54:23.883704 8545 tablet_loader.cc:96] loaded metadata for tablet c35e4ecbe2cc424089eaa5e0249df483 (table test-workload [id=7b7cc3ea430140598296ffa5c84d80d5])
I20250113 08:54:23.884160 8545 tablet_loader.cc:96] loaded metadata for tablet fe2a45627bfe4db0b9cdccfab81e2805 (table test-workload [id=7b7cc3ea430140598296ffa5c84d80d5])
I20250113 08:54:23.898135 8545 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 08:54:23.900472 8545 catalog_manager.cc:1260] Loaded cluster ID: 392e7551f3cc4c129852d2a40d11fb8b
I20250113 08:54:23.907285 8545 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 08:54:23.926683 8545 catalog_manager.cc:1505] Loading token signing keys...
I20250113 08:54:23.929239 8545 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 9fa3213b689141d9b07577d30495cd75: Loaded TSK: 0
I20250113 08:54:23.941751 8545 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250113 08:54:23.943910 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20250113 08:54:23.987828 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20250113 08:54:24.021914 8147 heartbeater.cc:346] Connected to a master server at 127.7.123.62:33371
W20250113 08:54:24.149559 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20250113 08:54:24.215611 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20250113 08:54:24.220870 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20250113 08:54:24.321230 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20250113 08:54:24.499264 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20250113 08:54:24.517830 8406 heartbeater.cc:346] Connected to a master server at 127.7.123.62:33371
I20250113 08:54:24.521600 8496 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" instance_seqno: 1736758455534695) as {username='slave'} at 127.7.123.4:56009; Asking this server to re-register.
I20250113 08:54:24.527704 8406 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:24.528075 8406 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:24.533162 8496 ts_manager.cc:194] Registered new tserver with Master: 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355)
W20250113 08:54:24.586100 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20250113 08:54:24.712496 8496 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1abc86bb580b4d14876623243c9d6594" instance_seqno: 1736758450814319) as {username='slave'} at 127.7.123.2:57025; Asking this server to re-register.
I20250113 08:54:24.713752 8016 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:24.714103 8016 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:24.720965 8496 ts_manager.cc:194] Registered new tserver with Master: 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
W20250113 08:54:24.722281 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20250113 08:54:24.742194 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20250113 08:54:24.764787 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20250113 08:54:24.915207 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20250113 08:54:25.002032 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20250113 08:54:25.036356 8496 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" instance_seqno: 1736758451341180) as {username='slave'} at 127.7.123.3:33459; Asking this server to re-register.
I20250113 08:54:25.046639 8147 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:25.047137 8147 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:25.059906 8496 ts_manager.cc:194] Registered new tserver with Master: 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
W20250113 08:54:25.141009 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20250113 08:54:25.275866 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250113 08:54:25.281915 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250113 08:54:25.289877 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20250113 08:54:25.422763 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20250113 08:54:25.564191 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250113 08:54:25.672027 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250113 08:54:25.747721 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250113 08:54:25.812261 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250113 08:54:25.826200 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250113 08:54:25.948933 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250113 08:54:26.080353 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250113 08:54:26.222894 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250113 08:54:26.265067 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250113 08:54:26.304814 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250113 08:54:26.415273 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250113 08:54:26.507789 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250113 08:54:26.565178 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20250113 08:54:26.684119 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.1:46439
--local_ip_for_outbound_sockets=127.7.123.1
--tserver_master_addrs=127.7.123.62:33371
--webserver_port=36711
--webserver_interface=127.7.123.1
--builtin_ntp_servers=127.7.123.20:43147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
W20250113 08:54:26.740913 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250113 08:54:26.816859 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250113 08:54:26.845659 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250113 08:54:27.001752 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250113 08:54:27.112576 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250113 08:54:27.171300 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250113 08:54:27.291532 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250113 08:54:27.334107 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20250113 08:54:27.435042 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20250113 08:54:27.610574 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250113 08:54:27.632225 8562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:27.632773 8562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:27.632953 8562 flags.cc:425] Enabled unsafe flag: --enable_log_gc=false
W20250113 08:54:27.633097 8562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:27.646631 8562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:27.647038 8562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.1
I20250113 08:54:27.664117 8562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:43147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.1:46439
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.123.1
--webserver_port=36711
--enable_log_gc=false
--tserver_master_addrs=127.7.123.62:33371
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.1
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:27.671173 8562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:27.675542 8562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:27.717057 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250113 08:54:27.727375 8570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:27.735954 8569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:27.742616 8562 server_base.cc:1034] running on GCE node
W20250113 08:54:27.744801 8572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:27.745908 8562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:27.750943 8562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:27.755676 8562 hybrid_clock.cc:648] HybridClock initialized: now 1736758467755600 us; error 74 us; skew 500 ppm
I20250113 08:54:27.756234 8562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:27.774245 8562 webserver.cc:458] Webserver started at http://127.7.123.1:36711/ using document root <none> and password file <none>
I20250113 08:54:27.776085 8562 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:27.776399 8562 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250113 08:54:27.786567 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20250113 08:54:27.828536 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20250113 08:54:27.830246 8562 fs_manager.cc:714] Time spent opening directory manager: real 0.044s user 0.018s sys 0.005s
I20250113 08:54:27.843273 8579 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:27.848083 8562 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s
I20250113 08:54:27.848506 8562 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal
uuid: "aea19a9cd21144508fdcfd7b3570a0e1"
format_stamp: "Formatted at 2025-01-13 08:54:10 on dist-test-slave-vxbw"
I20250113 08:54:27.850136 8562 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250113 08:54:27.878990 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20250113 08:54:27.995883 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20250113 08:54:28.029083 8562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:28.030508 8562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:28.031164 8562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:28.034183 8562 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:28.070205 8586 ts_tablet_manager.cc:542] Loading tablet metadata (0/6 complete)
I20250113 08:54:28.099124 8562 ts_tablet_manager.cc:579] Loaded tablet metadata (6 total tablets, 6 live tablets)
I20250113 08:54:28.099253 8562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.039s user 0.005s sys 0.003s
I20250113 08:54:28.099404 8562 ts_tablet_manager.cc:594] Registering tablets (0/6 complete)
I20250113 08:54:28.115876 8586 tablet_bootstrap.cc:492] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
I20250113 08:54:28.140848 8562 ts_tablet_manager.cc:610] Registered 6 tablets
I20250113 08:54:28.141079 8562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.042s user 0.014s sys 0.007s
W20250113 08:54:28.228658 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20250113 08:54:28.255558 8586 log.cc:826] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Log is configured to *not* fsync() on all Append() calls
W20250113 08:54:28.307104 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20250113 08:54:28.315093 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20250113 08:54:28.503968 8037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111) [suppressed 176 similar messages]
W20250113 08:54:28.521688 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20250113 08:54:28.655985 8586 tablet_bootstrap.cc:492] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=204 overwritten=0 applied=200 ignored=0} inserts{seen=1668 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20250113 08:54:28.657183 8586 tablet_bootstrap.cc:492] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap complete.
I20250113 08:54:28.661404 8586 ts_tablet_manager.cc:1397] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 0.554s user 0.377s sys 0.062s
I20250113 08:54:28.679272 8586 raft_consensus.cc:357] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Replica starting. Triggering 4 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
W20250113 08:54:28.692368 7906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20250113 08:54:28.693255 8586 raft_consensus.cc:738] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:28.694720 8586 consensus_queue.cc:260] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 200, Last appended: 1.204, Last appended by leader: 204, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
W20250113 08:54:28.694988 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20250113 08:54:28.696573 7905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.123.3:46625, user_credentials={real_user=slave}} blocked reactor thread for 105511us
I20250113 08:54:28.703480 8586 ts_tablet_manager.cc:1428] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.042s user 0.013s sys 0.015s
I20250113 08:54:28.704409 8586 tablet_bootstrap.cc:492] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
W20250113 08:54:28.704862 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20250113 08:54:28.886983 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20250113 08:54:28.962584 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20250113 08:54:28.982265 8612 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:28.982574 8612 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:28.989178 8612 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:29.079440 7968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
W20250113 08:54:29.153306 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20250113 08:54:29.155395 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:29.156143 8581 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:29.157214 8612 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:29.161219 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20250113 08:54:29.262110 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20250113 08:54:29.288359 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20250113 08:54:29.490417 8586 tablet_bootstrap.cc:492] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=202 overwritten=0 applied=200 ignored=0} inserts{seen=1679 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 08:54:29.492036 8586 tablet_bootstrap.cc:492] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap complete.
I20250113 08:54:29.494554 8586 ts_tablet_manager.cc:1397] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 0.790s user 0.335s sys 0.064s
I20250113 08:54:29.497002 8586 raft_consensus.cc:357] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:29.525485 8586 raft_consensus.cc:738] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:29.526078 8586 consensus_queue.cc:260] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 200, Last appended: 1.202, Last appended by leader: 202, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:29.526916 8586 ts_tablet_manager.cc:1428] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.032s user 0.014s sys 0.012s
I20250113 08:54:29.527400 8586 tablet_bootstrap.cc:492] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
I20250113 08:54:29.596642 8612 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:29.596892 8612 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:29.597759 8612 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:29.605952 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:29.612138 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:29.619776 8581 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:29.621387 8612 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:29.641757 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20250113 08:54:29.642609 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20250113 08:54:29.783593 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20250113 08:54:29.785214 8038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.123.2:45341, user_credentials={real_user=slave}} blocked reactor thread for 54271.8us
W20250113 08:54:29.835932 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20250113 08:54:29.898533 8612 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:29.898773 8612 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:29.899626 8612 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:29.900220 7968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:29.910660 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:29.933068 8581 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:29.933656 8612 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:29.955734 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20250113 08:54:30.061568 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20250113 08:54:30.120364 8586 tablet_bootstrap.cc:492] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=204 overwritten=0 applied=200 ignored=0} inserts{seen=1753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20250113 08:54:30.121802 8586 tablet_bootstrap.cc:492] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap complete.
I20250113 08:54:30.124282 8586 ts_tablet_manager.cc:1397] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 0.597s user 0.347s sys 0.056s
I20250113 08:54:30.125536 8586 raft_consensus.cc:357] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Replica starting. Triggering 4 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:30.181558 8586 raft_consensus.cc:738] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:30.182076 8586 consensus_queue.cc:260] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 200, Last appended: 1.204, Last appended by leader: 204, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:30.183362 8586 ts_tablet_manager.cc:1428] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.059s user 0.026s sys 0.009s
I20250113 08:54:30.183830 8586 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
W20250113 08:54:30.201856 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20250113 08:54:30.236593 8612 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:30.236836 8612 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:30.237600 8612 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:30.246551 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:30.252903 8562 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.1:46439
I20250113 08:54:30.252913 8710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.1:46439 every 8 connection(s)
I20250113 08:54:30.255442 8562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
I20250113 08:54:30.258915 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 8562
I20250113 08:54:30.262178 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:30.265089 8581 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:30.265755 8612 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:30.346323 8612 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:30.346557 8612 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:30.347393 8612 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:30.348884 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:30.353163 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 202 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:30.383497 8581 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:30.384130 8612 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:30.467490 8612 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:30.469101 8612 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:30.472689 8612 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:30.473584 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:30.492985 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 204 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:30.499966 8581 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:30.500634 8612 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:30.502669 8711 heartbeater.cc:346] Connected to a master server at 127.7.123.62:33371
I20250113 08:54:30.503315 8711 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:30.522262 8711 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:30.539772 8496 ts_manager.cc:194] Registered new tserver with Master: aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439)
I20250113 08:54:30.543913 8496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.1:54375
I20250113 08:54:30.732134 8658 raft_consensus.cc:3054] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:54:30.743810 8657 raft_consensus.cc:3054] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:54:30.745297 8658 pending_rounds.cc:77] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Aborting all ops after (but not including) 202
I20250113 08:54:30.745823 8658 pending_rounds.cc:99] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Aborting uncommitted WRITE_OP operation due to leader change: 1.203
I20250113 08:54:30.774251 8711 heartbeater.cc:502] Master 127.7.123.62:33371 was elected leader, sending a full tablet report...
I20250113 08:54:30.777439 8657 pending_rounds.cc:77] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Aborting all ops after (but not including) 201
I20250113 08:54:30.783005 8657 pending_rounds.cc:99] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Aborting uncommitted WRITE_OP operation due to leader change: 1.202
I20250113 08:54:30.783495 8657 pending_rounds.cc:99] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Aborting uncommitted WRITE_OP operation due to leader change: 1.203
I20250113 08:54:30.784034 8658 pending_rounds.cc:99] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1: Aborting uncommitted WRITE_OP operation due to leader change: 1.204
I20250113 08:54:30.794209 8657 pending_rounds.cc:99] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1: Aborting uncommitted WRITE_OP operation due to leader change: 1.204
W20250113 08:54:30.852175 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
W20250113 08:54:30.880545 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 111: this message will repeat every 5th retry.
W20250113 08:54:31.399127 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 111: this message will repeat every 5th retry.
W20250113 08:54:32.403110 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250113 08:54:32.448397 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 116: this message will repeat every 5th retry.
W20250113 08:54:32.759657 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 116: this message will repeat every 5th retry.
I20250113 08:54:32.990168 8612 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:32.990422 8612 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:32.995168 8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.3:55187 (request call id 1) took 2413 ms. Trace:
I20250113 08:54:32.995172 8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:37056 (request call id 7) took 2267 ms. Trace:
I20250113 08:54:32.999589 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 743 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:33.007478 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 743 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:33.014897 8581 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:33.015738 8743 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:33.004935 8656 rpcz_store.cc:276] 0113 08:54:30.697493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:30.697556 (+ 63us) service_pool.cc:225] Handling call
0113 08:54:32.962953 (+2265397us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x616000019954. stack: 00007f4647092fd6 00007f4647092d90 00007f464607248d 000000000096d544 000000000096d50d 00007f4655c30e1d 00007f4655c30f13 00007f4652939211 00007f4656630c3e 00007f4650cfdcb6 00007f4650cfd9e5 00007f464ddfa30e 00007f464ddf91ee 00007f464ddff425 00007f464ddffd4f 00007f464ddffb78
0113 08:54:32.964639 (+ 1686us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5199446912}
I20250113 08:54:33.018507 8612 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:33.073271 8658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.3:55187 (request call id 2) took 2496 ms. Trace:
I20250113 08:54:33.072844 8733 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:33.113296 8733 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:33.139895 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:33.140014 8733 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:33.145867 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:33.151913 8581 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:33.152900 8733 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:33.248682 8732 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:33.248914 8732 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:33.250106 8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.3:55187 (request call id 3) took 2632 ms. Trace:
I20250113 08:54:33.261260 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:33.266207 8732 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:33.271453 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:33.283607 8581 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:33.284225 8732 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:33.407456 8732 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:33.407727 8732 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:33.412104 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 743 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:33.418206 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 743 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:33.419509 8732 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:33.431056 8581 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:33.432632 8732 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:33.572532 8732 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:33.572757 8732 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:33.575255 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:33.577546 8732 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:33.582042 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:33.588629 8583 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:33.607066 8732 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:33.733728 8732 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:33.734028 8732 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:33.735945 8732 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:33.736796 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:33.741592 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:33.746543 8583 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:33.747179 8732 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:34.027467 8732 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:34.028151 8732 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:34.029096 8732 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:34.029887 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 743 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:34.037776 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 743 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:34.046269 8583 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:34.046929 8732 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:34.127846 8586 log.cc:926] Time spent T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
I20250113 08:54:34.250209 8732 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:34.250449 8732 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:34.266716 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:34.267334 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:34.268994 8583 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:34.272013 8732 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:34.272294 8732 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:33.073477 8658 rpcz_store.cc:276] 0113 08:54:30.577206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:30.577280 (+ 74us) service_pool.cc:225] Handling call
0113 08:54:30.708851 (+131571us) raft_consensus.cc:1466] Updating replica for 542 ops
0113 08:54:30.764549 (+ 55698us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:54:30.764558 (+ 9us) write_op.cc:454] Released schema lock
0113 08:54:30.784268 (+ 19710us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:54:30.784271 (+ 3us) write_op.cc:454] Released schema lock
0113 08:54:30.784639 (+ 368us) raft_consensus.cc:1529] Early marking committed up to index 202
0113 08:54:30.841290 (+ 56651us) raft_consensus.cc:1534] Triggering prepare for 542 ops
0113 08:54:30.841514 (+ 224us) write_op.cc:270] Start()
0113 08:54:30.841546 (+ 32us) write_op.cc:276] Timestamp: P: 1736758458777635 usec, L: 0
0113 08:54:30.849727 (+ 8181us) write_op.cc:270] Start()
0113 08:54:30.849784 (+ 57us) write_op.cc:276] Timestamp: P: 1736758458803543 usec, L: 0
0113 08:54:30.850126 (+ 342us) write_op.cc:270] Start()
0113 08:54:30.850156 (+ 30us) write_op.cc:276] Timestamp: P: 1736758458806258 usec, L: 0
0113 08:54:30.850320 (+ 164us) write_op.cc:270] Start()
0113 08:54:30.850345 (+ 25us) write_op.cc:276] Timestamp: P: 1736758458821210 usec, L: 0
0113 08:54:30.850571 (+ 226us) write_op.cc:270] Start()
0113 08:54:30.850604 (+ 33us) write_op.cc:276] Timestamp: P: 1736758458836491 usec, L: 0
0113 08:54:30.850827 (+ 223us) write_op.cc:270] Start()
0113 08:54:30.850849 (+ 22us) write_op.cc:276] Timestamp: P: 1736758458856434 usec, L: 0
0113 08:54:30.851007 (+ 158us) write_op.cc:270] Start()
0113 08:54:30.851025 (+ 18us) write_op.cc:276] Timestamp: P: 1736758458861068 usec, L: 0
0113 08:54:30.851193 (+ 168us) write_op.cc:270] Start()
0113 08:54:30.851212 (+ 19us) write_op.cc:276] Timestamp: P: 1736758458894625 usec, L: 0
0113 08:54:30.851418 (+ 206us) write_op.cc:270] Start()
0113 08:54:30.851445 (+ 27us) write_op.cc:276] Timestamp: P: 1736758458903116 usec, L: 0
0113 08:54:30.851633 (+ 188us) write_op.cc:270] Start()
0113 08:54:30.851653 (+ 20us) write_op.cc:276] Timestamp: P: 1736758458944601 usec, L: 0
0113 08:54:30.851845 (+ 192us) write_op.cc:270] Start()
0113 08:54:30.851866 (+ 21us) write_op.cc:276] Timestamp: P: 1736758458945624 usec, L: 0
0113 08:54:30.852072 (+ 206us) write_op.cc:270] Start()
0113 08:54:30.852093 (+ 21us) write_op.cc:276] Timestamp: P: 1736758458995511 usec, L: 0
0113 08:54:30.852303 (+ 210us) write_op.cc:270] Start()
0113 08:54:30.852325 (+ 22us) write_op.cc:276] Timestamp: P: 1736758458998796 usec, L: 0
0113 08:54:30.853037 (+ 712us) write_op.cc:270] Start()
0113 08:54:30.853066 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459000064 usec, L: 0
0113 08:54:30.853469 (+ 403us) write_op.cc:270] Start()
0113 08:54:30.853503 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459001467 usec, L: 0
0113 08:54:30.853701 (+ 198us) write_op.cc:270] Start()
0113 08:54:30.853734 (+ 33us) write_op.cc:276] Timestamp: P: 1736758459044873 usec, L: 0
0113 08:54:30.853982 (+ 248us) write_op.cc:270] Start()
0113 08:54:30.854006 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459049198 usec, L: 0
0113 08:54:30.854219 (+ 213us) write_op.cc:270] Start()
0113 08:54:30.854243 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459052916 usec, L: 0
0113 08:54:30.854421 (+ 178us) write_op.cc:270] Start()
0113 08:54:30.854447 (+ 26us) write_op.cc:276] Timestamp: P: 1736758459057014 usec, L: 0
0113 08:54:30.903229 (+ 48782us) write_op.cc:270] Start()
0113 08:54:30.903278 (+ 49us) write_op.cc:276] Timestamp: P: 1736758459101925 usec, L: 0
0113 08:54:30.921537 (+ 18259us) write_op.cc:270] Start()
0113 08:54:30.921566 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459106555 usec, L: 0
0113 08:54:30.923942 (+ 2376us) write_op.cc:270] Start()
0113 08:54:30.924018 (+ 76us) write_op.cc:276] Timestamp: P: 1736758459113676 usec, L: 0
0113 08:54:30.934920 (+ 10902us) write_op.cc:270] Start()
0113 08:54:30.934977 (+ 57us) write_op.cc:276] Timestamp: P: 1736758459117232 usec, L: 0
0113 08:54:30.935297 (+ 320us) write_op.cc:270] Start()
0113 08:54:30.935327 (+ 30us) write_op.cc:276] Timestamp: P: 1736758459135455 usec, L: 0
0113 08:54:30.935463 (+ 136us) write_op.cc:270] Start()
0113 08:54:30.935480 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459167174 usec, L: 0
0113 08:54:30.937275 (+ 1795us) write_op.cc:270] Start()
0113 08:54:30.937305 (+ 30us) write_op.cc:276] Timestamp: P: 1736758459185127 usec, L: 0
0113 08:54:30.952766 (+ 15461us) write_op.cc:270] Start()
0113 08:54:30.952791 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459191554 usec, L: 0
0113 08:54:30.963648 (+ 10857us) write_op.cc:270] Start()
0113 08:54:30.963689 (+ 41us) write_op.cc:276] Timestamp: P: 1736758459193153 usec, L: 0
0113 08:54:30.964845 (+ 1156us) write_op.cc:270] Start()
0113 08:54:30.964883 (+ 38us) write_op.cc:276] Timestamp: P: 1736758459232969 usec, L: 0
0113 08:54:30.968333 (+ 3450us) write_op.cc:270] Start()
0113 08:54:30.968359 (+ 26us) write_op.cc:276] Timestamp: P: 1736758459233645 usec, L: 0
0113 08:54:30.971705 (+ 3346us) write_op.cc:270] Start()
0113 08:54:30.971731 (+ 26us) write_op.cc:276] Timestamp: P: 1736758459240338 usec, L: 0
0113 08:54:30.973336 (+ 1605us) write_op.cc:270] Start()
0113 08:54:30.973371 (+ 35us) write_op.cc:276] Timestamp: P: 1736758459260476 usec, L: 0
0113 08:54:30.988781 (+ 15410us) write_op.cc:270] Start()
0113 08:54:30.988827 (+ 46us) write_op.cc:276] Timestamp: P: 1736758459290224 usec, L: 0
0113 08:54:30.998369 (+ 9542us) write_op.cc:270] Start()
0113 08:54:30.998395 (+ 26us) write_op.cc:276] Timestamp: P: 1736758459293666 usec, L: 0
0113 08:54:30.998601 (+ 206us) write_op.cc:270] Start()
0113 08:54:30.998622 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459315562 usec, L: 0
0113 08:54:31.007939 (+ 9317us) write_op.cc:270] Start()
0113 08:54:31.007999 (+ 60us) write_op.cc:276] Timestamp: P: 1736758459317729 usec, L: 0
0113 08:54:31.015156 (+ 7157us) write_op.cc:270] Start()
0113 08:54:31.015208 (+ 52us) write_op.cc:276] Timestamp: P: 1736758459337999 usec, L: 0
0113 08:54:31.024345 (+ 9137us) write_op.cc:270] Start()
0113 08:54:31.024375 (+ 30us) write_op.cc:276] Timestamp: P: 1736758459359413 usec, L: 0
0113 08:54:31.042055 (+ 17680us) write_op.cc:270] Start()
0113 08:54:31.042117 (+ 62us) write_op.cc:276] Timestamp: P: 1736758459361450 usec, L: 0
0113 08:54:31.045451 (+ 3334us) write_op.cc:270] Start()
0113 08:54:31.045490 (+ 39us) write_op.cc:276] Timestamp: P: 1736758459386861 usec, L: 0
0113 08:54:31.052177 (+ 6687us) write_op.cc:270] Start()
0113 08:54:31.052226 (+ 49us) write_op.cc:276] Timestamp: P: 1736758459413765 usec, L: 0
0113 08:54:31.056445 (+ 4219us) write_op.cc:270] Start()
0113 08:54:31.056468 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459417096 usec, L: 0
0113 08:54:31.056622 (+ 154us) write_op.cc:270] Start()
0113 08:54:31.056642 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459425623 usec, L: 0
0113 08:54:31.058344 (+ 1702us) write_op.cc:270] Start()
0113 08:54:31.058369 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459465038 usec, L: 0
0113 08:54:31.078480 (+ 20111us) write_op.cc:270] Start()
0113 08:54:31.081177 (+ 2697us) write_op.cc:276] Timestamp: P: 1736758459494476 usec, L: 0
0113 08:54:31.081967 (+ 790us) write_op.cc:270] Start()
0113 08:54:31.084056 (+ 2089us) write_op.cc:276] Timestamp: P: 1736758459496523 usec, L: 0
0113 08:54:31.090924 (+ 6868us) write_op.cc:270] Start()
0113 08:54:31.090982 (+ 58us) write_op.cc:276] Timestamp: P: 1736758459511515 usec, L: 0
0113 08:54:31.099446 (+ 8464us) write_op.cc:270] Start()
0113 08:54:31.099488 (+ 42us) write_op.cc:276] Timestamp: P: 1736758459515423 usec, L: 0
0113 08:54:31.124472 (+ 24984us) write_op.cc:270] Start()
0113 08:54:31.124504 (+ 32us) write_op.cc:276] Timestamp: P: 1736758459547970 usec, L: 0
0113 08:54:31.127870 (+ 3366us) write_op.cc:270] Start()
0113 08:54:31.127913 (+ 43us) write_op.cc:276] Timestamp: P: 1736758459549154 usec, L: 0
0113 08:54:31.135939 (+ 8026us) write_op.cc:270] Start()
0113 08:54:31.135968 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459576864 usec, L: 0
0113 08:54:31.136172 (+ 204us) write_op.cc:270] Start()
0113 08:54:31.136198 (+ 26us) write_op.cc:276] Timestamp: P: 1736758459590898 usec, L: 0
0113 08:54:31.136373 (+ 175us) write_op.cc:270] Start()
0113 08:54:31.136400 (+ 27us) write_op.cc:276] Timestamp: P: 1736758459593900 usec, L: 0
0113 08:54:31.136567 (+ 167us) write_op.cc:270] Start()
0113 08:54:31.136590 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459607651 usec, L: 0
0113 08:54:31.136753 (+ 163us) write_op.cc:270] Start()
0113 08:54:31.136775 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459640384 usec, L: 0
0113 08:54:31.136964 (+ 189us) write_op.cc:270] Start()
0113 08:54:31.136987 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459648141 usec, L: 0
0113 08:54:31.137142 (+ 155us) write_op.cc:270] Start()
0113 08:54:31.137164 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459662406 usec, L: 0
0113 08:54:31.137340 (+ 176us) write_op.cc:270] Start()
0113 08:54:31.137370 (+ 30us) write_op.cc:276] Timestamp: P: 1736758459666474 usec, L: 0
0113 08:54:31.137563 (+ 193us) write_op.cc:270] Start()
0113 08:54:31.137585 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459697596 usec, L: 0
0113 08:54:31.137957 (+ 372us) write_op.cc:270] Start()
0113 08:54:31.137980 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459740282 usec, L: 0
0113 08:54:31.138137 (+ 157us) write_op.cc:270] Start()
0113 08:54:31.138179 (+ 42us) write_op.cc:276] Timestamp: P: 1736758459741059 usec, L: 0
0113 08:54:31.138325 (+ 146us) write_op.cc:270] Start()
0113 08:54:31.138345 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459745624 usec, L: 0
0113 08:54:31.138485 (+ 140us) write_op.cc:270] Start()
0113 08:54:31.138503 (+ 18us) write_op.cc:276] Timestamp: P: 1736758459746628 usec, L: 0
0113 08:54:31.138656 (+ 153us) write_op.cc:270] Start()
0113 08:54:31.138679 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459792112 usec, L: 0
0113 08:54:31.138953 (+ 274us) write_op.cc:270] Start()
0113 08:54:31.138977 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459793010 usec, L: 0
0113 08:54:31.139151 (+ 174us) write_op.cc:270] Start()
0113 08:54:31.139178 (+ 27us) write_op.cc:276] Timestamp: P: 1736758459795620 usec, L: 0
0113 08:54:31.139331 (+ 153us) write_op.cc:270] Start()
0113 08:54:31.139355 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459806143 usec, L: 0
0113 08:54:31.139851 (+ 496us) write_op.cc:270] Start()
0113 08:54:31.139875 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459838445 usec, L: 0
0113 08:54:31.140040 (+ 165us) write_op.cc:270] Start()
0113 08:54:31.140064 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459841424 usec, L: 0
0113 08:54:31.140214 (+ 150us) write_op.cc:270] Start()
0113 08:54:31.140233 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459865634 usec, L: 0
0113 08:54:31.140361 (+ 128us) write_op.cc:270] Start()
0113 08:54:31.140380 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459866383 usec, L: 0
0113 08:54:31.140561 (+ 181us) write_op.cc:270] Start()
0113 08:54:31.140585 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459912724 usec, L: 0
0113 08:54:31.140716 (+ 131us) write_op.cc:270] Start()
0113 08:54:31.140733 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459914300 usec, L: 0
0113 08:54:31.140872 (+ 139us) write_op.cc:270] Start()
0113 08:54:31.140889 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459933893 usec, L: 0
0113 08:54:31.141049 (+ 160us) write_op.cc:270] Start()
0113 08:54:31.141070 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459940599 usec, L: 0
0113 08:54:31.141212 (+ 142us) write_op.cc:270] Start()
0113 08:54:31.141230 (+ 18us) write_op.cc:276] Timestamp: P: 1736758459986818 usec, L: 0
0113 08:54:31.142089 (+ 859us) write_op.cc:270] Start()
0113 08:54:31.142120 (+ 31us) write_op.cc:276] Timestamp: P: 1736758459990948 usec, L: 0
0113 08:54:31.142291 (+ 171us) write_op.cc:270] Start()
0113 08:54:31.142316 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459993827 usec, L: 0
0113 08:54:31.142460 (+ 144us) write_op.cc:270] Start()
0113 08:54:31.142515 (+ 55us) write_op.cc:276] Timestamp: P: 1736758460008612 usec, L: 0
0113 08:54:31.169935 (+ 27420us) write_op.cc:270] Start()
0113 08:54:31.169976 (+ 41us) write_op.cc:276] Timestamp: P: 1736758460034339 usec, L: 0
0113 08:54:31.178478 (+ 8502us) write_op.cc:270] Start()
0113 08:54:31.178534 (+ 56us) write_op.cc:276] Timestamp: P: 1736758460040416 usec, L: 0
0113 08:54:31.185074 (+ 6540us) write_op.cc:270] Start()
0113 08:54:31.185102 (+ 28us) write_op.cc:276] Timestamp: P: 1736758460049648 usec, L: 0
0113 08:54:31.189089 (+ 3987us) write_op.cc:270] Start()
0113 08:54:31.189114 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460068254 usec, L: 0
0113 08:54:31.189608 (+ 494us) write_op.cc:270] Start()
0113 08:54:31.189632 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460104324 usec, L: 0
0113 08:54:31.192604 (+ 2972us) write_op.cc:270] Start()
0113 08:54:31.192629 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460105366 usec, L: 0
0113 08:54:31.194023 (+ 1394us) write_op.cc:270] Start()
0113 08:54:31.194054 (+ 31us) write_op.cc:276] Timestamp: P: 1736758460124680 usec, L: 0
0113 08:54:31.199514 (+ 5460us) write_op.cc:270] Start()
0113 08:54:31.199550 (+ 36us) write_op.cc:276] Timestamp: P: 1736758460132191 usec, L: 0
0113 08:54:31.202168 (+ 2618us) write_op.cc:270] Start()
0113 08:54:31.202214 (+ 46us) write_op.cc:276] Timestamp: P: 1736758460169299 usec, L: 0
0113 08:54:31.202977 (+ 763us) write_op.cc:270] Start()
0113 08:54:31.203010 (+ 33us) write_op.cc:276] Timestamp: P: 1736758460174340 usec, L: 0
0113 08:54:31.204038 (+ 1028us) write_op.cc:270] Start()
0113 08:54:31.204062 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460185715 usec, L: 0
0113 08:54:31.204797 (+ 735us) write_op.cc:270] Start()
0113 08:54:31.204824 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460186428 usec, L: 0
0113 08:54:31.216484 (+ 11660us) write_op.cc:270] Start()
0113 08:54:31.216517 (+ 33us) write_op.cc:276] Timestamp: P: 1736758460214997 usec, L: 0
0113 08:54:31.217558 (+ 1041us) write_op.cc:270] Start()
0113 08:54:31.224921 (+ 7363us) write_op.cc:276] Timestamp: P: 1736758460251841 usec, L: 0
0113 08:54:31.230215 (+ 5294us) write_op.cc:270] Start()
0113 08:54:31.230245 (+ 30us) write_op.cc:276] Timestamp: P: 1736758460253294 usec, L: 0
0113 08:54:31.233391 (+ 3146us) write_op.cc:270] Start()
0113 08:54:31.233419 (+ 28us) write_op.cc:276] Timestamp: P: 1736758460254405 usec, L: 0
0113 08:54:31.242880 (+ 9461us) write_op.cc:270] Start()
0113 08:54:31.242915 (+ 35us) write_op.cc:276] Timestamp: P: 1736758460287610 usec, L: 0
0113 08:54:31.243087 (+ 172us) write_op.cc:270] Start()
0113 08:54:31.243109 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460289200 usec, L: 0
0113 08:54:31.243272 (+ 163us) write_op.cc:270] Start()
0113 08:54:31.243292 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460304219 usec, L: 0
0113 08:54:31.243440 (+ 148us) write_op.cc:270] Start()
0113 08:54:31.243460 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460331653 usec, L: 0
0113 08:54:31.243598 (+ 138us) write_op.cc:270] Start()
0113 08:54:31.243618 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460337494 usec, L: 0
0113 08:54:31.243911 (+ 293us) write_op.cc:270] Start()
0113 08:54:31.243938 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460342175 usec, L: 0
0113 08:54:31.259807 (+ 15869us) write_op.cc:270] Start()
0113 08:54:31.259859 (+ 52us) write_op.cc:276] Timestamp: P: 1736758460349132 usec, L: 0
0113 08:54:31.260095 (+ 236us) write_op.cc:270] Start()
0113 08:54:31.260116 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460401886 usec, L: 0
0113 08:54:31.264345 (+ 4229us) write_op.cc:270] Start()
0113 08:54:31.264382 (+ 37us) write_op.cc:276] Timestamp: P: 1736758460416476 usec, L: 0
0113 08:54:31.269750 (+ 5368us) write_op.cc:270] Start()
0113 08:54:31.269802 (+ 52us) write_op.cc:276] Timestamp: P: 1736758460430190 usec, L: 0
0113 08:54:31.270062 (+ 260us) write_op.cc:270] Start()
0113 08:54:31.270083 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460431354 usec, L: 0
0113 08:54:31.272914 (+ 2831us) write_op.cc:270] Start()
0113 08:54:31.272943 (+ 29us) write_op.cc:276] Timestamp: P: 1736758460464159 usec, L: 0
0113 08:54:31.274336 (+ 1393us) write_op.cc:270] Start()
0113 08:54:31.274371 (+ 35us) write_op.cc:276] Timestamp: P: 1736758460486039 usec, L: 0
0113 08:54:31.274640 (+ 269us) write_op.cc:270] Start()
0113 08:54:31.274666 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460487457 usec, L: 0
0113 08:54:31.276712 (+ 2046us) write_op.cc:270] Start()
0113 08:54:31.276742 (+ 30us) write_op.cc:276] Timestamp: P: 1736758460500014 usec, L: 0
0113 08:54:31.281744 (+ 5002us) write_op.cc:270] Start()
0113 08:54:31.281781 (+ 37us) write_op.cc:276] Timestamp: P: 1736758460520902 usec, L: 0
0113 08:54:31.303587 (+ 21806us) write_op.cc:270] Start()
0113 08:54:31.303628 (+ 41us) write_op.cc:276] Timestamp: P: 1736758460568846 usec, L: 0
0113 08:54:31.316661 (+ 13033us) write_op.cc:270] Start()
0113 08:54:31.316702 (+ 41us) write_op.cc:276] Timestamp: P: 1736758460569599 usec, L: 0
0113 08:54:31.316947 (+ 245us) write_op.cc:270] Start()
0113 08:54:31.316975 (+ 28us) write_op.cc:276] Timestamp: P: 1736758460572073 usec, L: 0
0113 08:54:31.333971 (+ 16996us) write_op.cc:270] Start()
0113 08:54:31.334008 (+ 37us) write_op.cc:276] Timestamp: P: 1736758460597855 usec, L: 0
0113 08:54:31.334208 (+ 200us) write_op.cc:270] Start()
0113 08:54:31.334230 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460610540 usec, L: 0
0113 08:54:31.334388 (+ 158us) write_op.cc:270] Start()
0113 08:54:31.334407 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460626118 usec, L: 0
0113 08:54:31.336681 (+ 2274us) write_op.cc:270] Start()
0113 08:54:31.336714 (+ 33us) write_op.cc:276] Timestamp: P: 1736758460641945 usec, L: 0
0113 08:54:31.336925 (+ 211us) write_op.cc:270] Start()
0113 08:54:31.336949 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460653672 usec, L: 0
0113 08:54:31.337123 (+ 174us) write_op.cc:270] Start()
0113 08:54:31.337147 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460672943 usec, L: 0
0113 08:54:31.337323 (+ 176us) write_op.cc:270] Start()
0113 08:54:31.337347 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460707255 usec, L: 0
0113 08:54:31.337493 (+ 146us) write_op.cc:270] Start()
0113 08:54:31.337513 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460708690 usec, L: 0
0113 08:54:31.337670 (+ 157us) write_op.cc:270] Start()
0113 08:54:31.337690 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460717792 usec, L: 0
0113 08:54:31.337875 (+ 185us) write_op.cc:270] Start()
0113 08:54:31.337898 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460735254 usec, L: 0
0113 08:54:31.338075 (+ 177us) write_op.cc:270] Start()
0113 08:54:31.338096 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460765094 usec, L: 0
0113 08:54:31.338251 (+ 155us) write_op.cc:270] Start()
0113 08:54:31.338272 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460771915 usec, L: 0
0113 08:54:31.338424 (+ 152us) write_op.cc:270] Start()
0113 08:54:31.338444 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460774759 usec, L: 0
0113 08:54:31.338609 (+ 165us) write_op.cc:270] Start()
0113 08:54:31.338629 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460788215 usec, L: 0
0113 08:54:31.352750 (+ 14121us) write_op.cc:270] Start()
0113 08:54:31.352780 (+ 30us) write_op.cc:276] Timestamp: P: 1736758460831364 usec, L: 0
0113 08:54:31.355879 (+ 3099us) write_op.cc:270] Start()
0113 08:54:31.355911 (+ 32us) write_op.cc:276] Timestamp: P: 1736758460835397 usec, L: 0
0113 08:54:31.356089 (+ 178us) write_op.cc:270] Start()
0113 08:54:31.356107 (+ 18us) write_op.cc:276] Timestamp: P: 1736758460836110 usec, L: 0
0113 08:54:31.357593 (+ 1486us) write_op.cc:270] Start()
0113 08:54:31.357630 (+ 37us) write_op.cc:276] Timestamp: P: 1736758460865858 usec, L: 0
0113 08:54:31.359169 (+ 1539us) write_op.cc:270] Start()
0113 08:54:31.359206 (+ 37us) write_op.cc:276] Timestamp: P: 1736758460876117 usec, L: 0
0113 08:54:31.360147 (+ 941us) write_op.cc:270] Start()
0113 08:54:31.360173 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460887036 usec, L: 0
0113 08:54:31.362548 (+ 2375us) write_op.cc:270] Start()
0113 08:54:31.362569 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460908378 usec, L: 0
0113 08:54:31.364106 (+ 1537us) write_op.cc:270] Start()
0113 08:54:31.364140 (+ 34us) write_op.cc:276] Timestamp: P: 1736758460932457 usec, L: 0
0113 08:54:31.371657 (+ 7517us) write_op.cc:270] Start()
0113 08:54:31.371691 (+ 34us) write_op.cc:276] Timestamp: P: 1736758460948434 usec, L: 0
0113 08:54:31.382930 (+ 11239us) write_op.cc:270] Start()
0113 08:54:31.382973 (+ 43us) write_op.cc:276] Timestamp: P: 1736758460966889 usec, L: 0
0113 08:54:31.383249 (+ 276us) write_op.cc:270] Start()
0113 08:54:31.383276 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460972842 usec, L: 0
0113 08:54:31.384892 (+ 1616us) write_op.cc:270] Start()
0113 08:54:31.384911 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460997250 usec, L: 0
0113 08:54:31.385037 (+ 126us) write_op.cc:270] Start()
0113 08:54:31.385053 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461012485 usec, L: 0
0113 08:54:31.388811 (+ 3758us) write_op.cc:270] Start()
0113 08:54:31.388837 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461021863 usec, L: 0
0113 08:54:31.389785 (+ 948us) write_op.cc:270] Start()
0113 08:54:31.389813 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461026061 usec, L: 0
0113 08:54:31.390876 (+ 1063us) write_op.cc:270] Start()
0113 08:54:31.390907 (+ 31us) write_op.cc:276] Timestamp: P: 1736758461064806 usec, L: 0
0113 08:54:31.404938 (+ 14031us) write_op.cc:270] Start()
0113 08:54:31.404979 (+ 41us) write_op.cc:276] Timestamp: P: 1736758461066289 usec, L: 0
0113 08:54:31.418914 (+ 13935us) write_op.cc:270] Start()
0113 08:54:31.418940 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461093764 usec, L: 0
0113 08:54:31.466281 (+ 47341us) spinlock_profiling.cc:243] Waited 47 ms on lock 0x60f000016ef8. stack: 00007f4647092fd6 00007f4647092d90 00007f464607248d 000000000096d544 000000000096d50d 000000000096ceda 00007f464fdf4848 00007f4652974629 00007f465291a2ea 00007f4652917b47 00007f46566220cd 00007f4650cf5b56 00007f4650cf5885 00007f464ddfa30e 00007f464ddf91ee 00007f464ddff425
0113 08:54:31.466685 (+ 404us) write_op.cc:270] Start()
0113 08:54:31.466730 (+ 45us) write_op.cc:276] Timestamp: P: 1736758461094590 usec, L: 0
0113 08:54:31.467018 (+ 288us) write_op.cc:270] Start()
0113 08:54:31.467046 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461123825 usec, L: 0
0113 08:54:31.467233 (+ 187us) write_op.cc:270] Start()
0113 08:54:31.467256 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461129545 usec, L: 0
0113 08:54:31.467430 (+ 174us) write_op.cc:270] Start()
0113 08:54:31.467453 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461159749 usec, L: 0
0113 08:54:31.467610 (+ 157us) write_op.cc:270] Start()
0113 08:54:31.467630 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461175345 usec, L: 0
0113 08:54:31.467797 (+ 167us) write_op.cc:270] Start()
0113 08:54:31.467820 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461179964 usec, L: 0
0113 08:54:31.468041 (+ 221us) write_op.cc:270] Start()
0113 08:54:31.468066 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461184494 usec, L: 0
0113 08:54:31.468221 (+ 155us) write_op.cc:270] Start()
0113 08:54:31.468240 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461220770 usec, L: 0
0113 08:54:31.468380 (+ 140us) write_op.cc:270] Start()
0113 08:54:31.468402 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461230382 usec, L: 0
0113 08:54:31.468553 (+ 151us) write_op.cc:270] Start()
0113 08:54:31.468577 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461231231 usec, L: 0
0113 08:54:31.468770 (+ 193us) write_op.cc:270] Start()
0113 08:54:31.468793 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461242624 usec, L: 0
0113 08:54:31.468960 (+ 167us) write_op.cc:270] Start()
0113 08:54:31.468986 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461274100 usec, L: 0
0113 08:54:31.469135 (+ 149us) write_op.cc:270] Start()
0113 08:54:31.469157 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461290512 usec, L: 0
0113 08:54:31.469304 (+ 147us) write_op.cc:270] Start()
0113 08:54:31.469324 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461297308 usec, L: 0
0113 08:54:31.469511 (+ 187us) write_op.cc:270] Start()
0113 08:54:31.469578 (+ 67us) write_op.cc:276] Timestamp: P: 1736758461301754 usec, L: 0
0113 08:54:31.469769 (+ 191us) write_op.cc:270] Start()
0113 08:54:31.469795 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461340956 usec, L: 0
0113 08:54:31.469965 (+ 170us) write_op.cc:270] Start()
0113 08:54:31.469988 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461341966 usec, L: 0
0113 08:54:31.470136 (+ 148us) write_op.cc:270] Start()
0113 08:54:31.470157 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461351835 usec, L: 0
0113 08:54:31.470342 (+ 185us) write_op.cc:270] Start()
0113 08:54:31.470367 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461376247 usec, L: 0
0113 08:54:31.470572 (+ 205us) write_op.cc:270] Start()
0113 08:54:31.470596 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461401870 usec, L: 0
0113 08:54:31.477848 (+ 7252us) write_op.cc:270] Start()
0113 08:54:31.477978 (+ 130us) write_op.cc:276] Timestamp: P: 1736758461404094 usec, L: 0
0113 08:54:31.478216 (+ 238us) write_op.cc:270] Start()
0113 08:54:31.478242 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461409981 usec, L: 0
0113 08:54:31.478405 (+ 163us) write_op.cc:270] Start()
0113 08:54:31.478424 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461438254 usec, L: 0
0113 08:54:31.478570 (+ 146us) write_op.cc:270] Start()
0113 08:54:31.478592 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461474285 usec, L: 0
0113 08:54:31.478788 (+ 196us) write_op.cc:270] Start()
0113 08:54:31.489068 (+ 10280us) write_op.cc:276] Timestamp: P: 1736758461489054 usec, L: 0
0113 08:54:31.489379 (+ 311us) write_op.cc:270] Start()
0113 08:54:31.489412 (+ 33us) write_op.cc:276] Timestamp: P: 1736758461493321 usec, L: 0
0113 08:54:31.489887 (+ 475us) write_op.cc:270] Start()
0113 08:54:31.489928 (+ 41us) write_op.cc:276] Timestamp: P: 1736758461495575 usec, L: 0
0113 08:54:31.501130 (+ 11202us) write_op.cc:270] Start()
0113 08:54:31.501172 (+ 42us) write_op.cc:276] Timestamp: P: 1736758461544068 usec, L: 0
0113 08:54:31.501401 (+ 229us) write_op.cc:270] Start()
0113 08:54:31.501427 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461544738 usec, L: 0
0113 08:54:31.501613 (+ 186us) write_op.cc:270] Start()
0113 08:54:31.501638 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461545712 usec, L: 0
0113 08:54:31.501817 (+ 179us) write_op.cc:270] Start()
0113 08:54:31.501839 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461576868 usec, L: 0
0113 08:54:31.508072 (+ 6233us) write_op.cc:270] Start()
0113 08:54:31.508107 (+ 35us) write_op.cc:276] Timestamp: P: 1736758461600914 usec, L: 0
0113 08:54:31.510610 (+ 2503us) write_op.cc:270] Start()
0113 08:54:31.510653 (+ 43us) write_op.cc:276] Timestamp: P: 1736758461644553 usec, L: 0
0113 08:54:31.510989 (+ 336us) write_op.cc:270] Start()
0113 08:54:31.511021 (+ 32us) write_op.cc:276] Timestamp: P: 1736758461647139 usec, L: 0
0113 08:54:31.511216 (+ 195us) write_op.cc:270] Start()
0113 08:54:31.511241 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461652610 usec, L: 0
0113 08:54:31.511435 (+ 194us) write_op.cc:270] Start()
0113 08:54:31.511460 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461676666 usec, L: 0
0113 08:54:31.511622 (+ 162us) write_op.cc:270] Start()
0113 08:54:31.511647 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461709996 usec, L: 0
0113 08:54:31.511802 (+ 155us) write_op.cc:270] Start()
0113 08:54:31.511825 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461710690 usec, L: 0
0113 08:54:31.512013 (+ 188us) write_op.cc:270] Start()
0113 08:54:31.512035 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461726198 usec, L: 0
0113 08:54:31.512180 (+ 145us) write_op.cc:270] Start()
0113 08:54:31.512198 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461745035 usec, L: 0
0113 08:54:31.512345 (+ 147us) write_op.cc:270] Start()
0113 08:54:31.512363 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461769005 usec, L: 0
0113 08:54:31.512525 (+ 162us) write_op.cc:270] Start()
0113 08:54:31.512544 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461780735 usec, L: 0
0113 08:54:31.512709 (+ 165us) write_op.cc:270] Start()
0113 08:54:31.512729 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461787301 usec, L: 0
0113 08:54:31.512872 (+ 143us) write_op.cc:270] Start()
0113 08:54:31.512892 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461831591 usec, L: 0
0113 08:54:31.513046 (+ 154us) write_op.cc:270] Start()
0113 08:54:31.513065 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461856530 usec, L: 0
0113 08:54:31.513197 (+ 132us) write_op.cc:270] Start()
0113 08:54:31.513216 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461907590 usec, L: 0
0113 08:54:31.513380 (+
W20250113 08:54:34.360141 8583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.123.2:45341, user_credentials={real_user=slave}} blocked reactor thread for 89431.6us
I20250113 08:54:34.417845 8752 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:34.418148 8752 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:34.532265 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:34.537746 8752 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:34.570307 7967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 744 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:34.608067 8583 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:34.658231 8752 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:35.073683 8752 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:35.073985 8752 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:35.075346 7967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 743 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:35.081872 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 743 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:35.092595 8752 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:35.098387 8581 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:33.250335 8657 rpcz_store.cc:276] 0113 08:54:30.617708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:30.617836 (+ 128us) service_pool.cc:225] Handling call
0113 08:54:30.697000 (+ 79164us) raft_consensus.cc:1466] Updating replica for 543 ops
0113 08:54:30.783366 (+ 86366us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:54:30.783370 (+ 4us) write_op.cc:454] Released schema lock
0113 08:54:30.783722 (+ 352us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:54:30.783727 (+ 5us) write_op.cc:454] Released schema lock
0113 08:54:30.794693 (+ 10966us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:54:30.794698 (+ 5us) write_op.cc:454] Released schema lock
0113 08:54:30.795267 (+ 569us) raft_consensus.cc:1529] Early marking committed up to index 201
0113 08:54:30.840117 (+ 44850us) raft_consensus.cc:1534] Triggering prepare for 543 ops
0113 08:54:30.840760 (+ 643us) write_op.cc:270] Start()
0113 08:54:30.840840 (+ 80us) write_op.cc:276] Timestamp: P: 1736758458752483 usec, L: 0
0113 08:54:30.851041 (+ 10201us) write_op.cc:270] Start()
0113 08:54:30.851066 (+ 25us) write_op.cc:276] Timestamp: P: 1736758458767139 usec, L: 0
0113 08:54:30.851235 (+ 169us) write_op.cc:270] Start()
0113 08:54:30.851258 (+ 23us) write_op.cc:276] Timestamp: P: 1736758458772566 usec, L: 0
0113 08:54:30.851443 (+ 185us) write_op.cc:270] Start()
0113 08:54:30.851468 (+ 25us) write_op.cc:276] Timestamp: P: 1736758458777658 usec, L: 0
0113 08:54:30.851610 (+ 142us) write_op.cc:270] Start()
0113 08:54:30.851627 (+ 17us) write_op.cc:276] Timestamp: P: 1736758458810980 usec, L: 0
0113 08:54:30.851832 (+ 205us) write_op.cc:270] Start()
0113 08:54:30.851860 (+ 28us) write_op.cc:276] Timestamp: P: 1736758458837977 usec, L: 0
0113 08:54:30.907771 (+ 55911us) write_op.cc:270] Start()
0113 08:54:30.907829 (+ 58us) write_op.cc:276] Timestamp: P: 1736758458858194 usec, L: 0
0113 08:54:30.924167 (+ 16338us) write_op.cc:270] Start()
0113 08:54:30.924185 (+ 18us) write_op.cc:276] Timestamp: P: 1736758458865396 usec, L: 0
0113 08:54:30.935604 (+ 11419us) write_op.cc:270] Start()
0113 08:54:30.935621 (+ 17us) write_op.cc:276] Timestamp: P: 1736758458890417 usec, L: 0
0113 08:54:30.936347 (+ 726us) write_op.cc:270] Start()
0113 08:54:30.936376 (+ 29us) write_op.cc:276] Timestamp: P: 1736758458901981 usec, L: 0
0113 08:54:30.938006 (+ 1630us) write_op.cc:270] Start()
0113 08:54:30.938033 (+ 27us) write_op.cc:276] Timestamp: P: 1736758458943508 usec, L: 0
0113 08:54:30.950973 (+ 12940us) write_op.cc:270] Start()
0113 08:54:30.951862 (+ 889us) write_op.cc:276] Timestamp: P: 1736758458949778 usec, L: 0
0113 08:54:30.952260 (+ 398us) write_op.cc:270] Start()
0113 08:54:30.952294 (+ 34us) write_op.cc:276] Timestamp: P: 1736758458994216 usec, L: 0
0113 08:54:30.952512 (+ 218us) write_op.cc:270] Start()
0113 08:54:30.952536 (+ 24us) write_op.cc:276] Timestamp: P: 1736758458997664 usec, L: 0
0113 08:54:30.973884 (+ 21348us) write_op.cc:270] Start()
0113 08:54:30.973923 (+ 39us) write_op.cc:276] Timestamp: P: 1736758458999104 usec, L: 0
0113 08:54:30.983586 (+ 9663us) write_op.cc:270] Start()
0113 08:54:30.983604 (+ 18us) write_op.cc:276] Timestamp: P: 1736758458999763 usec, L: 0
0113 08:54:30.987898 (+ 4294us) write_op.cc:270] Start()
0113 08:54:30.987981 (+ 83us) write_op.cc:276] Timestamp: P: 1736758459041357 usec, L: 0
0113 08:54:30.990460 (+ 2479us) write_op.cc:270] Start()
0113 08:54:30.990494 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459047234 usec, L: 0
0113 08:54:30.996982 (+ 6488us) write_op.cc:270] Start()
0113 08:54:31.001060 (+ 4078us) write_op.cc:276] Timestamp: P: 1736758459049897 usec, L: 0
0113 08:54:31.007164 (+ 6104us) write_op.cc:270] Start()
0113 08:54:31.007211 (+ 47us) write_op.cc:276] Timestamp: P: 1736758459058721 usec, L: 0
0113 08:54:31.013416 (+ 6205us) write_op.cc:270] Start()
0113 08:54:31.015388 (+ 1972us) write_op.cc:276] Timestamp: P: 1736758459084047 usec, L: 0
0113 08:54:31.018710 (+ 3322us) write_op.cc:270] Start()
0113 08:54:31.018755 (+ 45us) write_op.cc:276] Timestamp: P: 1736758459104520 usec, L: 0
0113 08:54:31.027224 (+ 8469us) write_op.cc:270] Start()
0113 08:54:31.027278 (+ 54us) write_op.cc:276] Timestamp: P: 1736758459111840 usec, L: 0
0113 08:54:31.031966 (+ 4688us) write_op.cc:270] Start()
0113 08:54:31.031990 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459113791 usec, L: 0
0113 08:54:31.042982 (+ 10992us) write_op.cc:270] Start()
0113 08:54:31.043016 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459133931 usec, L: 0
0113 08:54:31.043228 (+ 212us) write_op.cc:270] Start()
0113 08:54:31.043250 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459167896 usec, L: 0
0113 08:54:31.052466 (+ 9216us) write_op.cc:270] Start()
0113 08:54:31.052491 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459186880 usec, L: 0
0113 08:54:31.057682 (+ 5191us) write_op.cc:270] Start()
0113 08:54:31.057712 (+ 30us) write_op.cc:276] Timestamp: P: 1736758459193188 usec, L: 0
0113 08:54:31.061583 (+ 3871us) write_op.cc:270] Start()
0113 08:54:31.061862 (+ 279us) write_op.cc:276] Timestamp: P: 1736758459195033 usec, L: 0
0113 08:54:31.068954 (+ 7092us) write_op.cc:270] Start()
0113 08:54:31.068985 (+ 31us) write_op.cc:276] Timestamp: P: 1736758459232600 usec, L: 0
0113 08:54:31.069158 (+ 173us) write_op.cc:270] Start()
0113 08:54:31.069181 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459234457 usec, L: 0
0113 08:54:31.079181 (+ 10000us) write_op.cc:270] Start()
0113 08:54:31.079220 (+ 39us) write_op.cc:276] Timestamp: P: 1736758459239847 usec, L: 0
0113 08:54:31.121397 (+ 42177us) write_op.cc:270] Start()
0113 08:54:31.121439 (+ 42us) write_op.cc:276] Timestamp: P: 1736758459261090 usec, L: 0
0113 08:54:31.128200 (+ 6761us) write_op.cc:270] Start()
0113 08:54:31.128230 (+ 30us) write_op.cc:276] Timestamp: P: 1736758459293436 usec, L: 0
0113 08:54:31.153753 (+ 25523us) write_op.cc:270] Start()
0113 08:54:31.153802 (+ 49us) write_op.cc:276] Timestamp: P: 1736758459294295 usec, L: 0
0113 08:54:31.154073 (+ 271us) write_op.cc:270] Start()
0113 08:54:31.154100 (+ 27us) write_op.cc:276] Timestamp: P: 1736758459304714 usec, L: 0
0113 08:54:31.160445 (+ 6345us) write_op.cc:270] Start()
0113 08:54:31.160501 (+ 56us) write_op.cc:276] Timestamp: P: 1736758459314709 usec, L: 0
0113 08:54:31.165401 (+ 4900us) write_op.cc:270] Start()
0113 08:54:31.165425 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459337565 usec, L: 0
0113 08:54:31.166914 (+ 1489us) write_op.cc:270] Start()
0113 08:54:31.166943 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459359645 usec, L: 0
0113 08:54:31.171033 (+ 4090us) write_op.cc:270] Start()
0113 08:54:31.171069 (+ 36us) write_op.cc:276] Timestamp: P: 1736758459361399 usec, L: 0
0113 08:54:31.184241 (+ 13172us) write_op.cc:270] Start()
0113 08:54:31.184281 (+ 40us) write_op.cc:276] Timestamp: P: 1736758459407924 usec, L: 0
0113 08:54:31.184520 (+ 239us) write_op.cc:270] Start()
0113 08:54:31.184543 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459410850 usec, L: 0
0113 08:54:31.187269 (+ 2726us) write_op.cc:270] Start()
0113 08:54:31.187303 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459412993 usec, L: 0
0113 08:54:31.188628 (+ 1325us) write_op.cc:270] Start()
0113 08:54:31.188684 (+ 56us) write_op.cc:276] Timestamp: P: 1736758459416407 usec, L: 0
0113 08:54:31.188898 (+ 214us) write_op.cc:270] Start()
0113 08:54:31.188925 (+ 27us) write_op.cc:276] Timestamp: P: 1736758459477553 usec, L: 0
0113 08:54:31.191859 (+ 2934us) write_op.cc:270] Start()
0113 08:54:31.191888 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459492410 usec, L: 0
0113 08:54:31.192062 (+ 174us) write_op.cc:270] Start()
0113 08:54:31.194542 (+ 2480us) write_op.cc:276] Timestamp: P: 1736758459494177 usec, L: 0
0113 08:54:31.195167 (+ 625us) write_op.cc:270] Start()
0113 08:54:31.195188 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459514602 usec, L: 0
0113 08:54:31.195940 (+ 752us) write_op.cc:270] Start()
0113 08:54:31.195966 (+ 26us) write_op.cc:276] Timestamp: P: 1736758459516147 usec, L: 0
0113 08:54:31.196130 (+ 164us) write_op.cc:270] Start()
0113 08:54:31.196150 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459543692 usec, L: 0
0113 08:54:31.203883 (+ 7733us) write_op.cc:270] Start()
0113 08:54:31.203923 (+ 40us) write_op.cc:276] Timestamp: P: 1736758459544440 usec, L: 0
0113 08:54:31.205964 (+ 2041us) write_op.cc:270] Start()
0113 08:54:31.205984 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459578038 usec, L: 0
0113 08:54:31.219650 (+ 13666us) write_op.cc:270] Start()
0113 08:54:31.219685 (+ 35us) write_op.cc:276] Timestamp: P: 1736758459590550 usec, L: 0
0113 08:54:31.219849 (+ 164us) write_op.cc:270] Start()
0113 08:54:31.219865 (+ 16us) write_op.cc:276] Timestamp: P: 1736758459591599 usec, L: 0
0113 08:54:31.224749 (+ 4884us) write_op.cc:270] Start()
0113 08:54:31.224789 (+ 40us) write_op.cc:276] Timestamp: P: 1736758459602415 usec, L: 0
0113 08:54:31.227851 (+ 3062us) write_op.cc:270] Start()
0113 08:54:31.227887 (+ 36us) write_op.cc:276] Timestamp: P: 1736758459641152 usec, L: 0
0113 08:54:31.231791 (+ 3904us) write_op.cc:270] Start()
0113 08:54:31.231832 (+ 41us) write_op.cc:276] Timestamp: P: 1736758459648851 usec, L: 0
0113 08:54:31.232148 (+ 316us) write_op.cc:270] Start()
0113 08:54:31.232175 (+ 27us) write_op.cc:276] Timestamp: P: 1736758459661635 usec, L: 0
0113 08:54:31.232471 (+ 296us) write_op.cc:270] Start()
0113 08:54:31.232491 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459662397 usec, L: 0
0113 08:54:31.232600 (+ 109us) write_op.cc:270] Start()
0113 08:54:31.232613 (+ 13us) write_op.cc:276] Timestamp: P: 1736758459706694 usec, L: 0
0113 08:54:31.232728 (+ 115us) write_op.cc:270] Start()
0113 08:54:31.232745 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459739979 usec, L: 0
0113 08:54:31.232847 (+ 102us) write_op.cc:270] Start()
0113 08:54:31.232859 (+ 12us) write_op.cc:276] Timestamp: P: 1736758459741589 usec, L: 0
0113 08:54:31.232955 (+ 96us) write_op.cc:270] Start()
0113 08:54:31.232983 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459744061 usec, L: 0
0113 08:54:31.233151 (+ 168us) write_op.cc:270] Start()
0113 08:54:31.233166 (+ 15us) write_op.cc:276] Timestamp: P: 1736758459747417 usec, L: 0
0113 08:54:31.239613 (+ 6447us) write_op.cc:270] Start()
0113 08:54:31.239675 (+ 62us) write_op.cc:276] Timestamp: P: 1736758459790047 usec, L: 0
0113 08:54:31.239987 (+ 312us) write_op.cc:270] Start()
0113 08:54:31.240015 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459794456 usec, L: 0
0113 08:54:31.240212 (+ 197us) write_op.cc:270] Start()
0113 08:54:31.240236 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459797772 usec, L: 0
0113 08:54:31.240399 (+ 163us) write_op.cc:270] Start()
0113 08:54:31.240427 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459807993 usec, L: 0
0113 08:54:31.240669 (+ 242us) write_op.cc:270] Start()
0113 08:54:31.240693 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459836019 usec, L: 0
0113 08:54:31.240849 (+ 156us) write_op.cc:270] Start()
0113 08:54:31.240870 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459844498 usec, L: 0
0113 08:54:31.241027 (+ 157us) write_op.cc:270] Start()
0113 08:54:31.241044 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459864792 usec, L: 0
0113 08:54:31.241195 (+ 151us) write_op.cc:270] Start()
0113 08:54:31.241213 (+ 18us) write_op.cc:276] Timestamp: P: 1736758459866096 usec, L: 0
0113 08:54:31.241414 (+ 201us) write_op.cc:270] Start()
0113 08:54:31.241559 (+ 145us) write_op.cc:276] Timestamp: P: 1736758459907633 usec, L: 0
0113 08:54:31.241788 (+ 229us) write_op.cc:270] Start()
0113 08:54:31.241977 (+ 189us) write_op.cc:276] Timestamp: P: 1736758459914000 usec, L: 0
0113 08:54:31.271731 (+ 29754us) write_op.cc:270] Start()
0113 08:54:31.271768 (+ 37us) write_op.cc:276] Timestamp: P: 1736758459934631 usec, L: 0
0113 08:54:31.271963 (+ 195us) write_op.cc:270] Start()
0113 08:54:31.271983 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459935941 usec, L: 0
0113 08:54:31.278719 (+ 6736us) write_op.cc:270] Start()
0113 08:54:31.278753 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459985192 usec, L: 0
0113 08:54:31.280560 (+ 1807us) write_op.cc:270] Start()
0113 08:54:31.280596 (+ 36us) write_op.cc:276] Timestamp: P: 1736758459990349 usec, L: 0
0113 08:54:31.293940 (+ 13344us) write_op.cc:270] Start()
0113 08:54:31.296365 (+ 2425us) write_op.cc:276] Timestamp: P: 1736758459994010 usec, L: 0
0113 08:54:31.296705 (+ 340us) write_op.cc:270] Start()
0113 08:54:31.296736 (+ 31us) write_op.cc:276] Timestamp: P: 1736758460007750 usec, L: 0
0113 08:54:31.296919 (+ 183us) write_op.cc:270] Start()
0113 08:54:31.296945 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460037778 usec, L: 0
0113 08:54:31.297148 (+ 203us) write_op.cc:270] Start()
0113 08:54:31.297170 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460040001 usec, L: 0
0113 08:54:31.297316 (+ 146us) write_op.cc:270] Start()
0113 08:54:31.297336 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460062081 usec, L: 0
0113 08:54:31.297479 (+ 143us) write_op.cc:270] Start()
0113 08:54:31.297499 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460064677 usec, L: 0
0113 08:54:31.297657 (+ 158us) write_op.cc:270] Start()
0113 08:54:31.297678 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460106817 usec, L: 0
0113 08:54:31.298383 (+ 705us) write_op.cc:270] Start()
0113 08:54:31.298409 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460107628 usec, L: 0
0113 08:54:31.310504 (+ 12095us) write_op.cc:270] Start()
0113 08:54:31.310531 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460122678 usec, L: 0
0113 08:54:31.312832 (+ 2301us) write_op.cc:270] Start()
0113 08:54:31.312874 (+ 42us) write_op.cc:276] Timestamp: P: 1736758460131420 usec, L: 0
0113 08:54:31.314368 (+ 1494us) write_op.cc:270] Start()
0113 08:54:31.314394 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460173380 usec, L: 0
0113 08:54:31.327538 (+ 13144us) write_op.cc:270] Start()
0113 08:54:31.327589 (+ 51us) write_op.cc:276] Timestamp: P: 1736758460174811 usec, L: 0
0113 08:54:31.327847 (+ 258us) write_op.cc:270] Start()
0113 08:54:31.327873 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460182378 usec, L: 0
0113 08:54:31.328080 (+ 207us) write_op.cc:270] Start()
0113 08:54:31.328143 (+ 63us) write_op.cc:276] Timestamp: P: 1736758460190511 usec, L: 0
0113 08:54:31.328304 (+ 161us) write_op.cc:270] Start()
0113 08:54:31.328326 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460219865 usec, L: 0
0113 08:54:31.328709 (+ 383us) write_op.cc:270] Start()
0113 08:54:31.328735 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460230380 usec, L: 0
0113 08:54:31.328955 (+ 220us) write_op.cc:270] Start()
0113 08:54:31.328977 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460252191 usec, L: 0
0113 08:54:31.329161 (+ 184us) write_op.cc:270] Start()
0113 08:54:31.329188 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460253580 usec, L: 0
0113 08:54:31.329342 (+ 154us) write_op.cc:270] Start()
0113 08:54:31.329367 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460287600 usec, L: 0
0113 08:54:31.329556 (+ 189us) write_op.cc:270] Start()
0113 08:54:31.329581 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460289053 usec, L: 0
0113 08:54:31.329740 (+ 159us) write_op.cc:270] Start()
0113 08:54:31.329769 (+ 29us) write_op.cc:276] Timestamp: P: 1736758460301920 usec, L: 0
0113 08:54:31.329959 (+ 190us) write_op.cc:270] Start()
0113 08:54:31.329988 (+ 29us) write_op.cc:276] Timestamp: P: 1736758460333251 usec, L: 0
0113 08:54:31.330188 (+ 200us) write_op.cc:270] Start()
0113 08:54:31.330211 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460342032 usec, L: 0
0113 08:54:31.346865 (+ 16654us) write_op.cc:270] Start()
0113 08:54:31.346924 (+ 59us) write_op.cc:276] Timestamp: P: 1736758460342772 usec, L: 0
0113 08:54:31.347124 (+ 200us) write_op.cc:270] Start()
0113 08:54:31.347148 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460356219 usec, L: 0
0113 08:54:31.347305 (+ 157us) write_op.cc:270] Start()
0113 08:54:31.347325 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460399631 usec, L: 0
0113 08:54:31.347472 (+ 147us) write_op.cc:270] Start()
0113 08:54:31.347500 (+ 28us) write_op.cc:276] Timestamp: P: 1736758460400386 usec, L: 0
0113 08:54:31.347641 (+ 141us) write_op.cc:270] Start()
0113 08:54:31.347661 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460417291 usec, L: 0
0113 08:54:31.347811 (+ 150us) write_op.cc:270] Start()
0113 08:54:31.347832 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460425136 usec, L: 0
0113 08:54:31.347997 (+ 165us) write_op.cc:270] Start()
0113 08:54:31.348020 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460461624 usec, L: 0
0113 08:54:31.348174 (+ 154us) write_op.cc:270] Start()
0113 08:54:31.348194 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460485614 usec, L: 0
0113 08:54:31.348354 (+ 160us) write_op.cc:270] Start()
0113 08:54:31.348375 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460490361 usec, L: 0
0113 08:54:31.348522 (+ 147us) write_op.cc:270] Start()
0113 08:54:31.348541 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460497305 usec, L: 0
0113 08:54:31.348692 (+ 151us) write_op.cc:270] Start()
0113 08:54:31.348714 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460520079 usec, L: 0
0113 08:54:31.348873 (+ 159us) write_op.cc:270] Start()
0113 08:54:31.348893 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460569062 usec, L: 0
0113 08:54:31.349074 (+ 181us) write_op.cc:270] Start()
0113 08:54:31.349095 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460569879 usec, L: 0
0113 08:54:31.349261 (+ 166us) write_op.cc:270] Start()
0113 08:54:31.349283 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460572288 usec, L: 0
0113 08:54:31.349433 (+ 150us) write_op.cc:270] Start()
0113 08:54:31.349453 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460596607 usec, L: 0
0113 08:54:31.350667 (+ 1214us) write_op.cc:270] Start()
0113 08:54:31.350690 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460609973 usec, L: 0
0113 08:54:31.361311 (+ 10621us) write_op.cc:270] Start()
0113 08:54:31.361357 (+ 46us) write_op.cc:276] Timestamp: P: 1736758460639671 usec, L: 0
0113 08:54:31.361557 (+ 200us) write_op.cc:270] Start()
0113 08:54:31.361583 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460650341 usec, L: 0
0113 08:54:31.362150 (+ 567us) write_op.cc:270] Start()
0113 08:54:31.362170 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460651789 usec, L: 0
0113 08:54:31.366005 (+ 3835us) write_op.cc:270] Start()
0113 08:54:31.366039 (+ 34us) write_op.cc:276] Timestamp: P: 1736758460673330 usec, L: 0
0113 08:54:31.366255 (+ 216us) write_op.cc:270] Start()
0113 08:54:31.366276 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460707567 usec, L: 0
0113 08:54:31.371912 (+ 5636us) write_op.cc:270] Start()
0113 08:54:31.371929 (+ 17us) write_op.cc:276] Timestamp: P: 1736758460708144 usec, L: 0
0113 08:54:31.378844 (+ 6915us) write_op.cc:270] Start()
0113 08:54:31.378889 (+ 45us) write_op.cc:276] Timestamp: P: 1736758460733179 usec, L: 0
0113 08:54:31.384465 (+ 5576us) write_op.cc:270] Start()
0113 08:54:31.384496 (+ 31us) write_op.cc:276] Timestamp: P: 1736758460736128 usec, L: 0
0113 08:54:31.384694 (+ 198us) write_op.cc:270] Start()
0113 08:54:31.384716 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460764317 usec, L: 0
0113 08:54:31.388308 (+ 3592us) write_op.cc:270] Start()
0113 08:54:31.388339 (+ 31us) write_op.cc:276] Timestamp: P: 1736758460773261 usec, L: 0
0113 08:54:31.388563 (+ 224us) write_op.cc:270] Start()
0113 08:54:31.388588 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460774003 usec, L: 0
0113 08:54:31.391363 (+ 2775us) write_op.cc:270] Start()
0113 08:54:31.391393 (+ 30us) write_op.cc:276] Timestamp: P: 1736758460787944 usec, L: 0
0113 08:54:31.391560 (+ 167us) write_op.cc:270] Start()
0113 08:54:31.391578 (+ 18us) write_op.cc:276] Timestamp: P: 1736758460828506 usec, L: 0
0113 08:54:31.399369 (+ 7791us) write_op.cc:270] Start()
0113 08:54:31.399431 (+ 62us) write_op.cc:276] Timestamp: P: 1736758460831412 usec, L: 0
0113 08:54:31.407761 (+ 8330us) write_op.cc:270] Start()
0113 08:54:31.407805 (+ 44us) write_op.cc:276] Timestamp: P: 1736758460837734 usec, L: 0
0113 08:54:31.408065 (+ 260us) write_op.cc:270] Start()
0113 08:54:31.408090 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460868644 usec, L: 0
0113 08:54:31.408247 (+ 157us) write_op.cc:270] Start()
0113 08:54:31.408269 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460880961 usec, L: 0
0113 08:54:31.408425 (+ 156us) write_op.cc:270] Start()
0113 08:54:31.408443 (+ 18us) write_op.cc:276] Timestamp: P: 1736758460896781 usec, L: 0
0113 08:54:31.408576 (+ 133us) write_op.cc:270] Start()
0113 08:54:31.408593 (+ 17us) write_op.cc:276] Timestamp: P: 1736758460911181 usec, L: 0
0113 08:54:31.408767 (+ 174us) write_op.cc:270] Start()
0113 08:54:31.408786 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460930551 usec, L: 0
0113 08:54:31.408942 (+ 156us) write_op.cc:270] Start()
0113 08:54:31.408964 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460948507 usec, L: 0
0113 08:54:31.409114 (+ 150us) write_op.cc:270] Start()
0113 08:54:31.409131 (+ 17us) write_op.cc:276] Timestamp: P: 1736758460950396 usec, L: 0
0113 08:54:31.409251 (+ 120us) write_op.cc:270] Start()
0113 08:54:31.409266 (+ 15us) write_op.cc:276] Timestamp: P: 1736758460970629 usec, L: 0
0113 08:54:31.409377 (+ 111us) write_op.cc:270] Start()
0113 08:54:31.409391 (+ 14us) write_op.cc:276] Timestamp: P: 1736758461010895 usec, L: 0
0113 08:54:31.409519 (+ 128us) write_op.cc:270] Start()
0113 08:54:31.409535 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461025230 usec, L: 0
0113 08:54:31.409739 (+ 204us) write_op.cc:270] Start()
0113 08:54:31.409761 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461027790 usec, L: 0
0113 08:54:31.410101 (+ 340us) write_op.cc:270] Start()
0113 08:54:31.410141 (+ 40us) write_op.cc:276] Timestamp: P: 1736758461039505 usec, L: 0
0113 08:54:31.410374 (+ 233us) write_op.cc:270] Start()
0113 08:54:31.410397 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461066155 usec, L: 0
0113 08:54:31.416378 (+ 5981us) write_op.cc:270] Start()
0113 08:54:31.416423 (+ 45us) write_op.cc:276] Timestamp: P: 1736758461067605 usec, L: 0
0113 08:54:31.416715 (+ 292us) write_op.cc:270] Start()
0113 08:54:31.416743 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461095297 usec, L: 0
0113 08:54:31.416909 (+ 166us) write_op.cc:270] Start()
0113 08:54:31.416928 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461097993 usec, L: 0
0113 08:54:31.417063 (+ 135us) write_op.cc:270] Start()
0113 08:54:31.417081 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461114716 usec, L: 0
0113 08:54:31.417219 (+ 138us) write_op.cc:270] Start()
0113 08:54:31.417236 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461115511 usec, L: 0
0113 08:54:31.417363 (+ 127us) write_op.cc:270] Start()
0113 08:54:31.417381 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461177899 usec, L: 0
0113 08:54:31.417503 (+ 122us) write_op.cc:270] Start()
0113 08:54:31.417524 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461179982 usec, L: 0
0113 08:54:31.417694 (+ 170us) write_op.cc:270] Start()
0113 08:54:31.417715 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461183980 usec, L: 0
0113 08:54:31.417881 (+ 166us) write_op.cc:270] Start()
0113 08:54:31.417905 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461184724 usec, L: 0
0113 08:54:31.418081 (+ 176us) write_op.cc:270] Start()
0113 08:54:31.418104 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461222520 usec, L: 0
0113 08:54:31.418283 (+ 179us) write_op.cc:270] Start()
0113 08:54:31.418306 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461226240 usec, L: 0
0113 08:54:31.418498 (+ 192us) write_op.cc:270] Start()
0113 08:54:31.418522 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461230524 usec, L: 0
0113 08:54:31.418700 (+ 178us) write_op.cc:270] Start()
0113 08:54:31.418725 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461250922 usec, L: 0
0113 08:54:31.482673 (+ 63948us) write_op.cc:270] Start()
0113 08:54:31.482729 (+ 56us) write_op.cc:276] Timestamp: P: 1736758461273873 usec, L: 0
0113 08:54:31.483161 (+ 432us) write_op.cc:270] Start()
0113 08:54:31.483199 (+ 38us) write_op.cc:276] Timestamp: P: 1736758461291019 usec, L: 0
0113 08:54:31.483470 (+ 271us) write_op.cc:270] Start()
0113 08:54:31.483502 (+ 32us) write_op.cc:276] Timestamp: P: 1736758461292930 usec, L: 0
0113 08:54:31.483707 (+ 205us) write_op.cc:270] Start()
0113 08:54:31.483735 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461297559 usec, L: 0
0113 08:54:31.483926 (+ 191us) write_op.cc:270] Start()
0113 08:54:31.484024 (+ 98us) write_op.cc:276] Timestamp: P: 1736758461341792 usec, L: 0
0113 08:54:31.484249 (+ 225us) write_op.cc:270] Start()
0113 08:54:31.484270 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461343492 usec, L: 0
0113 08:54:31.484433 (+ 163us) write_op.cc:270] Start()
0113 08:54:31.484453 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461346445 usec, L: 0
0113 08:54:31.484595 (+ 142us) write_op.cc:270] Start()
0113 08:54:31.484615 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461378007 usec, L: 0
0113 08:54:31.484785 (+ 170us) write_op.cc:270] Start()
0113 08:54:31.484935 (+ 150us) write_op.cc:276] Timestamp: P: 1736758461404175 usec, L: 0
0113 08:54:31.485154 (+ 219us) write_op.cc:270] Start()
0113 08:54:31.485175 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461407494 usec, L: 0
0113 08:54:31.485371 (+ 196us) write_op.cc:270] Start()
0113 08:54:31.485393 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461411950 usec, L: 0
0113 08:54:31.485552 (+ 159us) write_op.cc:270] Start()
0113 08:54:31.485572 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461439075 usec, L: 0
0113 08:54:31.485725 (+ 153us) write_op.cc:270] Start()
0113 08:54:31.485747 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461476851 usec, L: 0
0113 08:54:31.485984 (+ 237us) write_op.cc:270] Start()
0113 08:54:31.486008 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461491528 usec, L: 0
0113 08:54:31.486187 (+ 179us) write_op.cc:270] Start()
0113 08:54:31.486209 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461494838 usec, L: 0
0113 08:54:31.486353 (+ 144us) write_op.cc:270] Start()
0113 08:54:31.486369 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461496054 usec, L: 0
0113 08:54:31.486468 (+ 99us) write_op.cc:270] Start()
0113 08:54:31.486482 (+ 14us) write_op.cc:276] Timestamp: P: 1736758461543717 usec, L: 0
0113 08:54:31.486606 (+ 124us) write_op.cc:270] Start()
0113 08:54:31.486627 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461545418 usec, L: 0
0113 08:54:31.517803 (+ 31176us) write_op.cc:270] Start()
0113 08:54:31.517847 (+ 44us) write_op.cc:276] Timestamp: P: 1736758461547726 usec, L: 0
0113 08:54:31.518146 (+ 299us) write_op.cc:270] Start()
0113 08:54:31.518378 (+ 232us) write_op.cc:276] Timestamp: P: 1736758461579938 usec, L: 0
0113 08:54:31.518677 (+ 299us) write_op.cc:270] Start()
0113 08:54:31.518707 (+ 30us) write_op.cc:276] Timestamp: P: 1736758461601736 usec, L: 0
0113 08:54:31.518925 (+ 218us) write_op.cc:270] Start()
0113 08:54:31.518962 (+ 37us) write_op.cc:276] Timestamp: P: 1736758461643312 usec, L: 0
0113 08:54:31.519133 (+ 171us) write_op.cc:270] Start()
0113 08:54:31.519158 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461647149 usec, L: 0
0113 08:54:31.519324 (+ 166us) write_op.cc:270] Start()
0113 08:54:31.519347 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461655194 usec, L: 0
0113 08:54:31.519504 (+ 157us) write_op.cc:270] Start()
0113 08:54:31.519525 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461672051 usec, L: 0
0113 08:54:31.519655 (+ 130us) write_op.cc:270] Start()
0113 08:54:31.519670 (+ 15us) write_op.cc:276] Timestamp: P: 1736758461705996 usec, L: 0
0113 08:54:31.519852 (+ 182us) write_op.cc:270] Start()
0113 08:54:31.519874 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461710230 usec, L: 0
0113 08:54:31.520036 (+ 162us) write_op.cc:270] Start()
0113 08:54:31.520055 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461710791 usec, L: 0
0113 08:54:31.520177 (+ 122us) write_op.cc:270] Start()
0113 08:54:31.520192 (+ 15us) write_op.cc:276] Timestamp: P: 1736758461744622 usec, L: 0
0113 08:54:31.520287 (+ 95us) write_op.cc:270] Start()
0113 08:54:31.520300 (+ 13us) write_op.cc:276] Timestamp: P: 1736758461762169 usec, L: 0
0113 08:54:31.520390 (+ 90us) write_op.cc:270] Start()
0113 08:54:31.520402 (+ 12us) write_op.cc:276] Timestamp: P: 1736758461781572 usec, L: 0
0113 08:54:31.520527 (+ 125us) write_op.cc:270] Start()
0113 08:54:31.520548 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461787028 usec, L: 0
0113 08:54:31.520699 (+ 151us) write_op.cc:270] Start()
0113 08:54:31.520715 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461827958 usec, L: 0
0113 08:54:31.520807 (+ 92us) write_op.cc:270] Start()
0113 08:54:31.520820 (+ 13us) write_op.cc:276] Timestamp: P: 1736758461854900 usec, L: 0
0113 08:54:31.520932 (+ 112us) write_op.cc:270] Start()
0113 08:54:31.520963 (+ 31us) write_op.cc:276] Timestamp: P: 1736758461892369 usec, L: 0
0113 08:54:31.521119 (+ 156us) write_op.cc:270] Start()
0113 08:54:31.521139 (+ 20us
I20250113 08:54:35.208743 8754 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:35.369689 8726 mvcc.cc:204] Tried to move back new op lower bound from 7113762694167306240 to 7113762647035002880. Current Snapshot: MvccSnapshot[applied={T|T < 7113762667079892992}]
I20250113 08:54:35.393581 8586 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap replaying log segment 1/1 (109.1K/125.3K this segment, stats: ops{read=178 overwritten=0 applied=178 ignored=0} inserts{seen=1447 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:54:35.517539 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20250113 08:54:35.590750 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 121: this message will repeat every 5th retry.
I20250113 08:54:35.609269 8752 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:35.610831 8752 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:35.612860 8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.3:55187 (request call id 14) took 1190 ms. Trace:
I20250113 08:54:35.650470 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 918 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:35.650868 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 918 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:35.656716 8752 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:35.665472 8583 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:35.671435 8752 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:35.648615 8656 rpcz_store.cc:276] 0113 08:54:34.415830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:34.415917 (+ 87us) service_pool.cc:225] Handling call
0113 08:54:34.528019 (+112102us) raft_consensus.cc:1466] Updating replica for 174 ops
0113 08:54:34.774519 (+246500us) raft_consensus.cc:1529] Early marking committed up to index 744
0113 08:54:34.774647 (+ 128us) raft_consensus.cc:1534] Triggering prepare for 174 ops
0113 08:54:34.776419 (+ 1772us) write_op.cc:270] Start()
0113 08:54:34.776476 (+ 57us) write_op.cc:276] Timestamp: P: 1736758470341850 usec, L: 0
0113 08:54:34.776748 (+ 272us) write_op.cc:270] Start()
0113 08:54:34.776778 (+ 30us) write_op.cc:276] Timestamp: P: 1736758470385211 usec, L: 0
0113 08:54:34.776994 (+ 216us) write_op.cc:270] Start()
0113 08:54:34.777016 (+ 22us) write_op.cc:276] Timestamp: P: 1736758470386020 usec, L: 0
0113 08:54:34.777158 (+ 142us) write_op.cc:270] Start()
0113 08:54:34.777180 (+ 22us) write_op.cc:276] Timestamp: P: 1736758470403876 usec, L: 0
0113 08:54:34.777345 (+ 165us) write_op.cc:270] Start()
0113 08:54:34.777366 (+ 21us) write_op.cc:276] Timestamp: P: 1736758470453608 usec, L: 0
0113 08:54:34.777536 (+ 170us) write_op.cc:270] Start()
0113 08:54:34.777559 (+ 23us) write_op.cc:276] Timestamp: P: 1736758470483189 usec, L: 0
0113 08:54:34.777730 (+ 171us) write_op.cc:270] Start()
0113 08:54:34.777774 (+ 44us) write_op.cc:276] Timestamp: P: 1736758470516159 usec, L: 0
0113 08:54:34.777959 (+ 185us) write_op.cc:270] Start()
0113 08:54:34.777982 (+ 23us) write_op.cc:276] Timestamp: P: 1736758470524229 usec, L: 0
0113 08:54:34.778149 (+ 167us) write_op.cc:270] Start()
0113 08:54:34.778171 (+ 22us) write_op.cc:276] Timestamp: P: 1736758470539081 usec, L: 0
0113 08:54:34.778339 (+ 168us) write_op.cc:270] Start()
0113 08:54:34.778360 (+ 21us) write_op.cc:276] Timestamp: P: 1736758470587064 usec, L: 0
0113 08:54:34.778541 (+ 181us) write_op.cc:270] Start()
0113 08:54:34.778563 (+ 22us) write_op.cc:276] Timestamp: P: 1736758470625897 usec, L: 0
0113 08:54:34.778731 (+ 168us) write_op.cc:270] Start()
0113 08:54:34.778760 (+ 29us) write_op.cc:276] Timestamp: P: 1736758470633549 usec, L: 0
0113 08:54:34.791986 (+ 13226us) write_op.cc:270] Start()
0113 08:54:34.792033 (+ 47us) write_op.cc:276] Timestamp: P: 1736758470634263 usec, L: 0
0113 08:54:34.797679 (+ 5646us) write_op.cc:270] Start()
0113 08:54:34.797739 (+ 60us) write_op.cc:276] Timestamp: P: 1736758470669640 usec, L: 0
0113 08:54:34.803622 (+ 5883us) write_op.cc:270] Start()
0113 08:54:34.803661 (+ 39us) write_op.cc:276] Timestamp: P: 1736758470695979 usec, L: 0
0113 08:54:34.816337 (+ 12676us) write_op.cc:270] Start()
0113 08:54:34.816397 (+ 60us) write_op.cc:276] Timestamp: P: 1736758470699302 usec, L: 0
0113 08:54:34.816736 (+ 339us) write_op.cc:270] Start()
0113 08:54:34.832216 (+ 15480us) write_op.cc:276] Timestamp: P: 1736758470752077 usec, L: 0
0113 08:54:34.845822 (+ 13606us) write_op.cc:270] Start()
0113 08:54:34.845865 (+ 43us) write_op.cc:276] Timestamp: P: 1736758470779993 usec, L: 0
0113 08:54:34.856991 (+ 11126us) write_op.cc:270] Start()
0113 08:54:34.857044 (+ 53us) write_op.cc:276] Timestamp: P: 1736758470829950 usec, L: 0
0113 08:54:34.857296 (+ 252us) write_op.cc:270] Start()
0113 08:54:34.857316 (+ 20us) write_op.cc:276] Timestamp: P: 1736758470845492 usec, L: 0
0113 08:54:34.871634 (+ 14318us) write_op.cc:270] Start()
0113 08:54:34.871679 (+ 45us) write_op.cc:276] Timestamp: P: 1736758470847213 usec, L: 0
0113 08:54:34.873759 (+ 2080us) write_op.cc:270] Start()
0113 08:54:34.873798 (+ 39us) write_op.cc:276] Timestamp: P: 1736758470899543 usec, L: 0
0113 08:54:34.875621 (+ 1823us) write_op.cc:270] Start()
0113 08:54:34.875654 (+ 33us) write_op.cc:276] Timestamp: P: 1736758470912064 usec, L: 0
0113 08:54:34.875872 (+ 218us) write_op.cc:270] Start()
0113 08:54:34.875894 (+ 22us) write_op.cc:276] Timestamp: P: 1736758470912781 usec, L: 0
0113 08:54:34.907560 (+ 31666us) write_op.cc:270] Start()
0113 08:54:34.908229 (+ 669us) write_op.cc:276] Timestamp: P: 1736758470913485 usec, L: 0
0113 08:54:34.919770 (+ 11541us) write_op.cc:270] Start()
0113 08:54:34.919829 (+ 59us) write_op.cc:276] Timestamp: P: 1736758471002777 usec, L: 0
0113 08:54:34.920124 (+ 295us) write_op.cc:270] Start()
0113 08:54:34.920144 (+ 20us) write_op.cc:276] Timestamp: P: 1736758471029559 usec, L: 0
0113 08:54:34.920302 (+ 158us) write_op.cc:270] Start()
0113 08:54:34.920325 (+ 23us) write_op.cc:276] Timestamp: P: 1736758471034961 usec, L: 0
0113 08:54:34.920505 (+ 180us) write_op.cc:270] Start()
0113 08:54:34.920525 (+ 20us) write_op.cc:276] Timestamp: P: 1736758471042076 usec, L: 0
0113 08:54:34.920659 (+ 134us) write_op.cc:270] Start()
0113 08:54:34.920687 (+ 28us) write_op.cc:276] Timestamp: P: 1736758471071958 usec, L: 0
0113 08:54:34.920815 (+ 128us) write_op.cc:270] Start()
0113 08:54:34.920832 (+ 17us) write_op.cc:276] Timestamp: P: 1736758471117454 usec, L: 0
0113 08:54:34.920968 (+ 136us) write_op.cc:270] Start()
0113 08:54:34.920986 (+ 18us) write_op.cc:276] Timestamp: P: 1736758471147530 usec, L: 0
0113 08:54:34.933252 (+ 12266us) write_op.cc:270] Start()
0113 08:54:34.933299 (+ 47us) write_op.cc:276] Timestamp: P: 1736758471150275 usec, L: 0
0113 08:54:34.945656 (+ 12357us) write_op.cc:270] Start()
0113 08:54:34.945698 (+ 42us) write_op.cc:276] Timestamp: P: 1736758471153115 usec, L: 0
0113 08:54:34.951903 (+ 6205us) write_op.cc:270] Start()
0113 08:54:34.951939 (+ 36us) write_op.cc:276] Timestamp: P: 1736758471176628 usec, L: 0
0113 08:54:34.957052 (+ 5113us) write_op.cc:270] Start()
0113 08:54:34.957089 (+ 37us) write_op.cc:276] Timestamp: P: 1736758471237334 usec, L: 0
0113 08:54:34.962419 (+ 5330us) write_op.cc:270] Start()
0113 08:54:34.962471 (+ 52us) write_op.cc:276] Timestamp: P: 1736758471238507 usec, L: 0
0113 08:54:34.964512 (+ 2041us) write_op.cc:270] Start()
0113 08:54:34.968434 (+ 3922us) write_op.cc:276] Timestamp: P: 1736758471246455 usec, L: 0
0113 08:54:34.972587 (+ 4153us) write_op.cc:270] Start()
0113 08:54:34.972622 (+ 35us) write_op.cc:276] Timestamp: P: 1736758471256185 usec, L: 0
0113 08:54:34.978303 (+ 5681us) write_op.cc:270] Start()
0113 08:54:34.978337 (+ 34us) write_op.cc:276] Timestamp: P: 1736758471305866 usec, L: 0
0113 08:54:34.982473 (+ 4136us) write_op.cc:270] Start()
0113 08:54:34.982505 (+ 32us) write_op.cc:276] Timestamp: P: 1736758471307405 usec, L: 0
0113 08:54:34.983606 (+ 1101us) write_op.cc:270] Start()
0113 08:54:34.983645 (+ 39us) write_op.cc:276] Timestamp: P: 1736758471331993 usec, L: 0
0113 08:54:34.983877 (+ 232us) write_op.cc:270] Start()
0113 08:54:34.983897 (+ 20us) write_op.cc:276] Timestamp: P: 1736758471332523 usec, L: 0
0113 08:54:34.989191 (+ 5294us) write_op.cc:270] Start()
0113 08:54:34.989226 (+ 35us) write_op.cc:276] Timestamp: P: 1736758471402577 usec, L: 0
0113 08:54:34.989482 (+ 256us) write_op.cc:270] Start()
0113 08:54:34.989509 (+ 27us) write_op.cc:276] Timestamp: P: 1736758471404223 usec, L: 0
0113 08:54:34.989667 (+ 158us) write_op.cc:270] Start()
0113 08:54:34.989687 (+ 20us) write_op.cc:276] Timestamp: P: 1736758471423618 usec, L: 0
0113 08:54:34.990850 (+ 1163us) write_op.cc:270] Start()
0113 08:54:34.990879 (+ 29us) write_op.cc:276] Timestamp: P: 1736758471456515 usec, L: 0
0113 08:54:34.991062 (+ 183us) write_op.cc:270] Start()
0113 08:54:34.991083 (+ 21us) write_op.cc:276] Timestamp: P: 1736758471509147 usec, L: 0
0113 08:54:34.991262 (+ 179us) write_op.cc:270] Start()
0113 08:54:34.991282 (+ 20us) write_op.cc:276] Timestamp: P: 1736758471540097 usec, L: 0
0113 08:54:34.991487 (+ 205us) write_op.cc:270] Start()
0113 08:54:34.991508 (+ 21us) write_op.cc:276] Timestamp: P: 1736758471563585 usec, L: 0
0113 08:54:35.003807 (+ 12299us) write_op.cc:270] Start()
0113 08:54:35.003858 (+ 51us) write_op.cc:276] Timestamp: P: 1736758471578456 usec, L: 0
0113 08:54:35.018602 (+ 14744us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x616000019078. stack: 00007f4647092fd6 00007f4647092d90 00007f464607248d 000000000096d544 000000000096d50d 000000000096ceda 00007f4653ca13e3 00007f4653c8f9ba 00007f4653c3a86f 00007f4653c3992e 00007f46528fb6b1 00007f465291a208 00007f4652917b47 00007f46566220cd 00007f4650cf5b56 00007f4650cf5885
0113 08:54:35.018938 (+ 336us) write_op.cc:270] Start()
0113 08:54:35.018972 (+ 34us) write_op.cc:276] Timestamp: P: 1736758471590274 usec, L: 0
0113 08:54:35.019177 (+ 205us) write_op.cc:270] Start()
0113 08:54:35.019204 (+ 27us) write_op.cc:276] Timestamp: P: 1736758471651804 usec, L: 0
0113 08:54:35.019363 (+ 159us) write_op.cc:270] Start()
0113 08:54:35.019385 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471675135 usec, L: 0
0113 08:54:35.019594 (+ 209us) write_op.cc:270] Start()
0113 08:54:35.019615 (+ 21us) write_op.cc:276] Timestamp: P: 1736758471676887 usec, L: 0
0113 08:54:35.023594 (+ 3979us) write_op.cc:270] Start()
0113 08:54:35.023640 (+ 46us) write_op.cc:276] Timestamp: P: 1736758471677959 usec, L: 0
0113 08:54:35.023913 (+ 273us) write_op.cc:270] Start()
0113 08:54:35.023969 (+ 56us) write_op.cc:276] Timestamp: P: 1736758471695105 usec, L: 0
0113 08:54:35.024167 (+ 198us) write_op.cc:270] Start()
0113 08:54:35.024189 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471743448 usec, L: 0
0113 08:54:35.027125 (+ 2936us) write_op.cc:270] Start()
0113 08:54:35.027168 (+ 43us) write_op.cc:276] Timestamp: P: 1736758471753212 usec, L: 0
0113 08:54:35.027464 (+ 296us) write_op.cc:270] Start()
0113 08:54:35.027493 (+ 29us) write_op.cc:276] Timestamp: P: 1736758471757875 usec, L: 0
0113 08:54:35.027690 (+ 197us) write_op.cc:270] Start()
0113 08:54:35.027712 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471785494 usec, L: 0
0113 08:54:35.038257 (+ 10545us) write_op.cc:270] Start()
0113 08:54:35.038300 (+ 43us) write_op.cc:276] Timestamp: P: 1736758471822927 usec, L: 0
0113 08:54:35.038567 (+ 267us) write_op.cc:270] Start()
0113 08:54:35.038589 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471845289 usec, L: 0
0113 08:54:35.038785 (+ 196us) write_op.cc:270] Start()
0113 08:54:35.038840 (+ 55us) write_op.cc:276] Timestamp: P: 1736758471847776 usec, L: 0
0113 08:54:35.039091 (+ 251us) write_op.cc:270] Start()
0113 08:54:35.039118 (+ 27us) write_op.cc:276] Timestamp: P: 1736758471892242 usec, L: 0
0113 08:54:35.039294 (+ 176us) write_op.cc:270] Start()
0113 08:54:35.039316 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471942862 usec, L: 0
0113 08:54:35.039515 (+ 199us) write_op.cc:270] Start()
0113 08:54:35.039537 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471962504 usec, L: 0
0113 08:54:35.039701 (+ 164us) write_op.cc:270] Start()
0113 08:54:35.039722 (+ 21us) write_op.cc:276] Timestamp: P: 1736758471966224 usec, L: 0
0113 08:54:35.039905 (+ 183us) write_op.cc:270] Start()
0113 08:54:35.039928 (+ 23us) write_op.cc:276] Timestamp: P: 1736758471992728 usec, L: 0
0113 08:54:35.040102 (+ 174us) write_op.cc:270] Start()
0113 08:54:35.040149 (+ 47us) write_op.cc:276] Timestamp: P: 1736758472038693 usec, L: 0
0113 08:54:35.040309 (+ 160us) write_op.cc:270] Start()
0113 08:54:35.040337 (+ 28us) write_op.cc:276] Timestamp: P: 1736758472042752 usec, L: 0
0113 08:54:35.049435 (+ 9098us) write_op.cc:270] Start()
0113 08:54:35.049466 (+ 31us) write_op.cc:276] Timestamp: P: 1736758472047761 usec, L: 0
0113 08:54:35.049737 (+ 271us) write_op.cc:270] Start()
0113 08:54:35.049760 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472063472 usec, L: 0
0113 08:54:35.049919 (+ 159us) write_op.cc:270] Start()
0113 08:54:35.049944 (+ 25us) write_op.cc:276] Timestamp: P: 1736758472093986 usec, L: 0
0113 08:54:35.050124 (+ 180us) write_op.cc:270] Start()
0113 08:54:35.050146 (+ 22us) write_op.cc:276] Timestamp: P: 1736758472155846 usec, L: 0
0113 08:54:35.050285 (+ 139us) write_op.cc:270] Start()
0113 08:54:35.050303 (+ 18us) write_op.cc:276] Timestamp: P: 1736758472159483 usec, L: 0
0113 08:54:35.050478 (+ 175us) write_op.cc:270] Start()
0113 08:54:35.050510 (+ 32us) write_op.cc:276] Timestamp: P: 1736758472166828 usec, L: 0
0113 08:54:35.051786 (+ 1276us) write_op.cc:270] Start()
0113 08:54:35.051813 (+ 27us) write_op.cc:276] Timestamp: P: 1736758472188776 usec, L: 0
0113 08:54:35.052028 (+ 215us) write_op.cc:270] Start()
0113 08:54:35.052053 (+ 25us) write_op.cc:276] Timestamp: P: 1736758472216449 usec, L: 0
0113 08:54:35.052241 (+ 188us) write_op.cc:270] Start()
0113 08:54:35.052263 (+ 22us) write_op.cc:276] Timestamp: P: 1736758472244527 usec, L: 0
0113 08:54:35.052445 (+ 182us) write_op.cc:270] Start()
0113 08:54:35.052465 (+ 20us) write_op.cc:276] Timestamp: P: 1736758472249748 usec, L: 0
0113 08:54:35.052617 (+ 152us) write_op.cc:270] Start()
0113 08:54:35.052636 (+ 19us) write_op.cc:276] Timestamp: P: 1736758472279845 usec, L: 0
0113 08:54:35.052822 (+ 186us) write_op.cc:270] Start()
0113 08:54:35.052842 (+ 20us) write_op.cc:276] Timestamp: P: 1736758472297400 usec, L: 0
0113 08:54:35.052988 (+ 146us) write_op.cc:270] Start()
0113 08:54:35.053011 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472303862 usec, L: 0
0113 08:54:35.060181 (+ 7170us) write_op.cc:270] Start()
0113 08:54:35.060216 (+ 35us) write_op.cc:276] Timestamp: P: 1736758472322635 usec, L: 0
0113 08:54:35.060422 (+ 206us) write_op.cc:270] Start()
0113 08:54:35.074964 (+ 14542us) write_op.cc:276] Timestamp: P: 1736758472354958 usec, L: 0
0113 08:54:35.085275 (+ 10311us) write_op.cc:270] Start()
0113 08:54:35.085307 (+ 32us) write_op.cc:276] Timestamp: P: 1736758472369242 usec, L: 0
0113 08:54:35.093882 (+ 8575us) write_op.cc:270] Start()
0113 08:54:35.093923 (+ 41us) write_op.cc:276] Timestamp: P: 1736758472420001 usec, L: 0
0113 08:54:35.094170 (+ 247us) write_op.cc:270] Start()
0113 08:54:35.094191 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472425688 usec, L: 0
0113 08:54:35.107025 (+ 12834us) write_op.cc:270] Start()
0113 08:54:35.107062 (+ 37us) write_op.cc:276] Timestamp: P: 1736758472431056 usec, L: 0
0113 08:54:35.116437 (+ 9375us) write_op.cc:270] Start()
0113 08:54:35.116467 (+ 30us) write_op.cc:276] Timestamp: P: 1736758472433547 usec, L: 0
0113 08:54:35.126212 (+ 9745us) write_op.cc:270] Start()
0113 08:54:35.126249 (+ 37us) write_op.cc:276] Timestamp: P: 1736758472531160 usec, L: 0
0113 08:54:35.179385 (+ 53136us) write_op.cc:270] Start()
0113 08:54:35.179432 (+ 47us) write_op.cc:276] Timestamp: P: 1736758472537365 usec, L: 0
0113 08:54:35.179735 (+ 303us) write_op.cc:270] Start()
0113 08:54:35.179766 (+ 31us) write_op.cc:276] Timestamp: P: 1736758472542713 usec, L: 0
0113 08:54:35.189605 (+ 9839us) write_op.cc:270] Start()
0113 08:54:35.189641 (+ 36us) write_op.cc:276] Timestamp: P: 1736758472546187 usec, L: 0
0113 08:54:35.196037 (+ 6396us) write_op.cc:270] Start()
0113 08:54:35.196082 (+ 45us) write_op.cc:276] Timestamp: P: 1736758472623352 usec, L: 0
0113 08:54:35.210906 (+ 14824us) write_op.cc:270] Start()
0113 08:54:35.210943 (+ 37us) write_op.cc:276] Timestamp: P: 1736758472648570 usec, L: 0
0113 08:54:35.212508 (+ 1565us) write_op.cc:270] Start()
0113 08:54:35.212544 (+ 36us) write_op.cc:276] Timestamp: P: 1736758472649260 usec, L: 0
0113 08:54:35.215515 (+ 2971us) write_op.cc:270] Start()
0113 08:54:35.215546 (+ 31us) write_op.cc:276] Timestamp: P: 1736758472649907 usec, L: 0
0113 08:54:35.215786 (+ 240us) write_op.cc:270] Start()
0113 08:54:35.215809 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472711834 usec, L: 0
0113 08:54:35.227480 (+ 11671us) write_op.cc:270] Start()
0113 08:54:35.227509 (+ 29us) write_op.cc:276] Timestamp: P: 1736758472721593 usec, L: 0
0113 08:54:35.246892 (+ 19383us) write_op.cc:270] Start()
0113 08:54:35.246944 (+ 52us) write_op.cc:276] Timestamp: P: 1736758472727381 usec, L: 0
0113 08:54:35.247219 (+ 275us) write_op.cc:270] Start()
0113 08:54:35.247242 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472737572 usec, L: 0
0113 08:54:35.249154 (+ 1912us) write_op.cc:270] Start()
0113 08:54:35.249184 (+ 30us) write_op.cc:276] Timestamp: P: 1736758472803274 usec, L: 0
0113 08:54:35.251345 (+ 2161us) write_op.cc:270] Start()
0113 08:54:35.251387 (+ 42us) write_op.cc:276] Timestamp: P: 1736758472825202 usec, L: 0
0113 08:54:35.251743 (+ 356us) write_op.cc:270] Start()
0113 08:54:35.251766 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472826475 usec, L: 0
0113 08:54:35.259905 (+ 8139us) write_op.cc:270] Start()
0113 08:54:35.259944 (+ 39us) write_op.cc:276] Timestamp: P: 1736758472835732 usec, L: 0
0113 08:54:35.265780 (+ 5836us) write_op.cc:270] Start()
0113 08:54:35.265816 (+ 36us) write_op.cc:276] Timestamp: P: 1736758472931217 usec, L: 0
0113 08:54:35.268845 (+ 3029us) write_op.cc:270] Start()
0113 08:54:35.268876 (+ 31us) write_op.cc:276] Timestamp: P: 1736758472940589 usec, L: 0
0113 08:54:35.270921 (+ 2045us) write_op.cc:270] Start()
0113 08:54:35.270953 (+ 32us) write_op.cc:276] Timestamp: P: 1736758472953163 usec, L: 0
0113 08:54:35.271537 (+ 584us) write_op.cc:270] Start()
0113 08:54:35.271564 (+ 27us) write_op.cc:276] Timestamp: P: 1736758472960336 usec, L: 0
0113 08:54:35.272688 (+ 1124us) write_op.cc:270] Start()
0113 08:54:35.272714 (+ 26us) write_op.cc:276] Timestamp: P: 1736758472994559 usec, L: 0
0113 08:54:35.281935 (+ 9221us) write_op.cc:270] Start()
0113 08:54:35.281974 (+ 39us) write_op.cc:276] Timestamp: P: 1736758473037190 usec, L: 0
0113 08:54:35.282223 (+ 249us) write_op.cc:270] Start()
0113 08:54:35.282249 (+ 26us) write_op.cc:276] Timestamp: P: 1736758473043152 usec, L: 0
0113 08:54:35.285679 (+ 3430us) write_op.cc:270] Start()
0113 08:54:35.285707 (+ 28us) write_op.cc:276] Timestamp: P: 1736758473063081 usec, L: 0
0113 08:54:35.285901 (+ 194us) write_op.cc:270] Start()
0113 08:54:35.285922 (+ 21us) write_op.cc:276] Timestamp: P: 1736758473074351 usec, L: 0
0113 08:54:35.295347 (+ 9425us) write_op.cc:270] Start()
0113 08:54:35.295391 (+ 44us) write_op.cc:276] Timestamp: P: 1736758473090158 usec, L: 0
0113 08:54:35.303898 (+ 8507us) write_op.cc:270] Start()
0113 08:54:35.303939 (+ 41us) write_op.cc:276] Timestamp: P: 1736758473126586 usec, L: 0
0113 08:54:35.310988 (+ 7049us) write_op.cc:270] Start()
0113 08:54:35.311022 (+ 34us) write_op.cc:276] Timestamp: P: 1736758473128025 usec, L: 0
0113 08:54:35.313158 (+ 2136us) write_op.cc:270] Start()
0113 08:54:35.313189 (+ 31us) write_op.cc:276] Timestamp: P: 1736758473169921 usec, L: 0
0113 08:54:35.315535 (+ 2346us) write_op.cc:270] Start()
0113 08:54:35.315568 (+ 33us) write_op.cc:276] Timestamp: P: 1736758473179055 usec, L: 0
0113 08:54:35.315775 (+ 207us) write_op.cc:270] Start()
0113 08:54:35.315798 (+ 23us) write_op.cc:276] Timestamp: P: 1736758473226867 usec, L: 0
0113 08:54:35.315961 (+ 163us) write_op.cc:270] Start()
0113 08:54:35.315984 (+ 23us) write_op.cc:276] Timestamp: P: 1736758473230425 usec, L: 0
0113 08:54:35.316141 (+ 157us) write_op.cc:270] Start()
0113 08:54:35.316162 (+ 21us) write_op.cc:276] Timestamp: P: 1736758473231052 usec, L: 0
0113 08:54:35.317822 (+ 1660us) write_op.cc:270] Start()
0113 08:54:35.317847 (+ 25us) write_op.cc:276] Timestamp: P: 1736758473247583 usec, L: 0
0113 08:54:35.318021 (+ 174us) write_op.cc:270] Start()
0113 08:54:35.318042 (+ 21us) write_op.cc:276] Timestamp: P: 1736758473280715 usec, L: 0
0113 08:54:35.318180 (+ 138us) write_op.cc:270] Start()
0113 08:54:35.318199 (+ 19us) write_op.cc:276] Timestamp: P: 1736758473317968 usec, L: 0
0113 08:54:35.330507 (+ 12308us) write_op.cc:270] Start()
0113 08:54:35.331110 (+ 603us) write_op.cc:276] Timestamp: P: 1736758473322016 usec, L: 0
0113 08:54:35.333585 (+ 2475us) write_op.cc:270] Start()
0113 08:54:35.333619 (+ 34us) write_op.cc:276] Timestamp: P: 1736758473347541 usec, L: 0
0113 08:54:35.333874 (+ 255us) write_op.cc:270] Start()
0113 08:54:35.333898 (+ 24us) write_op.cc:276] Timestamp: P: 1736758473384583 usec, L: 0
0113 08:54:35.334085 (+ 187us) write_op.cc:270] Start()
0113 08:54:35.334105 (+ 20us) write_op.cc:276] Timestamp: P: 1736758473399131 usec, L: 0
0113 08:54:35.334613 (+ 508us) write_op.cc:270] Start()
0113 08:54:35.334639 (+ 26us) write_op.cc:276] Timestamp: P: 1736758473403496 usec, L: 0
0113 08:54:35.334854 (+ 215us) write_op.cc:270] Start()
0113 08:54:35.334878 (+ 24us) write_op.cc:276] Timestamp: P: 1736758473446773 usec, L: 0
0113 08:54:35.335044 (+ 166us) write_op.cc:270] Start()
0113 08:54:35.335065 (+ 21us) write_op.cc:276] Timestamp: P: 1736758473486709 usec, L: 0
0113 08:54:35.338299 (+ 3234us) write_op.cc:270] Start()
0113 08:54:35.338326 (+ 27us) write_op.cc:276] Timestamp: P: 1736758473487363 usec, L: 0
0113 08:54:35.384274 (+ 45948us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x616000019130. stack: 00007f4647092fd6 00007f4647092d90 00007f464607248d 000000000096d544 000000000096d50d 000000000096ceda 00007f4656559e9b 00007f4653c8efa7 00007f4653c3a86f 00007f4653c3992e 00007f46528fb6b1 00007f465291a208 00007f4652917b47 00007f46566220cd 00007f4650cf5b56 00007f4650cf5885
0113 08:54:35.384297 (+ 23us) write_op.cc:270] Start()
0113 08:54:35.384342 (+ 45us) write_op.cc:276] Timestamp: P: 1736758473502897 usec, L: 0
0113 08:54:35.389396 (+ 5054us) write_op.cc:270] Start()
0113 08:54:35.389444 (+ 48us) write_op.cc:276] Timestamp: P: 1736758473550732 usec, L: 0
0113 08:54:35.398721 (+ 9277us) write_op.cc:270] Start()
0113 08:54:35.398757 (+ 36us) write_op.cc:276] Timestamp: P: 1736758473612213 usec, L: 0
0113 08:54:35.404397 (+ 5640us) write_op.cc:270] Start()
0113 08:54:35.404437 (+ 40us) write_op.cc:276] Timestamp: P: 1736758473621181 usec, L: 0
0113 08:54:35.404677 (+ 240us) write_op.cc:270] Start()
0113 08:54:35.404700 (+ 23us) write_op.cc:276] Timestamp: P: 1736758473627376 usec, L: 0
0113 08:54:35.408126 (+ 3426us) write_op.cc:270] Start()
0113 08:54:35.408160 (+ 34us) write_op.cc:276] Timestamp: P: 1736758473638618 usec, L: 0
0113 08:54:35.412364 (+ 4204us) write_op.cc:270] Start()
0113 08:54:35.412402 (+ 38us) write_op.cc:276] Timestamp: P: 1736758473668970 usec, L: 0
0113 08:54:35.412646 (+ 244us) write_op.cc:270] Start()
0113 08:54:35.412666 (+ 20us) write_op.cc:276] Timestamp: P: 1736758473702129 usec, L: 0
0113 08:54:35.415685 (+ 3019us) write_op.cc:270] Start()
0113 08:54:35.415721 (+ 36us) write_op.cc:276] Timestamp: P: 1736758473709934 usec, L: 0
0113 08:54:35.421559 (+ 5838us) write_op.cc:270] Start()
0113 08:54:35.421598 (+ 39us) write_op.cc:276] Timestamp: P: 1736758473720168 usec, L: 0
0113 08:54:35.431760 (+ 10162us) write_op.cc:270] Start()
0113 08:54:35.431794 (+ 34us) write_op.cc:276] Timestamp: P: 1736758473735257 usec, L: 0
0113 08:54:35.437609 (+ 5815us) write_op.cc:270] Start()
0113 08:54:35.437640 (+ 31us) write_op.cc:276] Timestamp: P: 1736758473786930 usec, L: 0
0113 08:54:35.442866 (+ 5226us) write_op.cc:270] Start()
0113 08:54:35.442900 (+ 34us) write_op.cc:276] Timestamp: P: 1736758473789875 usec, L: 0
0113 08:54:35.453127 (+ 10227us) write_op.cc:270] Start()
0113 08:54:35.453166 (+ 39us) write_op.cc:276] Timestamp: P: 1736758473794228 usec, L: 0
0113 08:54:35.456126 (+ 2960us) write_op.cc:270] Start()
0113 08:54:35.456162 (+ 36us) write_op.cc:276] Timestamp: P: 1736758473799288 usec, L: 0
0113 08:54:35.478261 (+ 22099us) write_op.cc:270] Start()
0113 08:54:35.478295 (+ 34us) write_op.cc:276] Timestamp: P: 1736758473882026 usec, L: 0
0113 08:54:35.485770 (+ 7475us) write_op.cc:270] Start()
0113 08:54:35.485802 (+ 32us) write_op.cc:276] Timestamp: P: 1736758473886790 usec, L: 0
0113 08:54:35.495116 (+ 9314us) write_op.cc:270] Start()
0113 08:54:35.495141 (+ 25us) write_op.cc:276] Timestamp: P: 1736758473892178 usec, L: 0
0113 08:54:35.495307 (+ 166us) write_op.cc:270] Start()
0113 08:54:35.495324 (+ 17us) write_op.cc:276] Timestamp: P: 1736758473903283 usec, L: 0
0113 08:54:35.495449 (+ 125us) write_op.cc:270] Start()
0113 08:54:35.495467 (+ 18us) write_op.cc:276] Timestamp: P: 1736758473998346 usec, L: 0
0113 08:54:35.495592 (+ 125us) write_op.cc:270] Start()
0113 08:54:35.495608 (+ 16us) write_op.cc:276] Timestamp: P: 1736758474000426 usec, L: 0
0113 08:54:35.495755 (+ 147us) write_op.cc:270] Start()
0113 08:54:35.495771 (+ 16us) write_op.cc:276] Timestamp: P: 1736758474008339 usec, L: 0
0113 08:54:35.495889 (+ 118us) write_op.cc:270] Start()
0113 08:54:35.495906 (+ 17us) write_op.cc:276] Timestamp: P: 1736758474035582 usec, L: 0
0113 08:54:35.496032 (+ 126us) write_op.cc:270] Start()
0113 08:54:35.496048 (+ 16us) write_op.cc:276] Timestamp: P: 1736758474077876 usec, L: 0
0113 08:54:35.496163 (+ 115us) write_op.cc:270] Start()
0113 08:54:35.496179 (+ 16us) write_op.cc:276] Timestamp: P: 1736758474106142 usec, L: 0
0113 08:54:35.496343 (+ 164us) write_op.cc:270] Start()
0113 08:54:35.496362 (+ 19us) write_op.cc:276] Timestamp: P: 1736758474111532 usec, L: 0
0113 08:54:35.496480 (+ 118us) write_op.cc:270] Start()
0113 08:54:35.496495 (+ 15us) write_op.cc:276] Timestamp: P: 1736758474128367 usec, L: 0
0113 08:54:35.496607 (+ 112us) write_op.cc:270] Start()
0113 08:54:35.496623 (+ 16us) write_op.cc:276] Timestamp: P: 1736758474162250 usec, L: 0
0113 08:54:35.496745 (+ 122us) write_op.cc:270] Start()
0113 08:54:35.496760 (+ 15us) write_op.cc:276] Timestamp: P: 1736758474175676 usec, L: 0
0113 08:54:35.496879 (+ 119us) write_op.cc:270] Start()
0113 08:54:35.496894 (+ 15us) write_op.cc:276] Timestamp: P: 1736758474221065 usec, L: 0
0113 08:54:35.497000 (+ 106us) write_op.cc:270] Start()
0113 08:54:35.497017 (+ 17us) write_op.cc:276] Timestamp: P: 1736758474223899 usec, L: 0
0113 08:54:35.497130 (+ 113us) write_op.cc:270] Start()
0113 08:54:35.497146 (+ 16us) write_op.cc:276] Timestamp: P: 1736758474241194 usec, L: 0
0113 08:54:35.497250 (+ 104us) write_op.cc:270] Start()
0113 08:54:35.497265 (+ 15us) write_op.cc:276] Timestamp: P: 1736758474268507 usec, L: 0
0113 08:54:35.497387 (+ 122us) write_op.cc:270] Start()
0113 08:54:35.497404 (+ 17us) write_op.cc:276] Timestamp: P: 1736758474286150 usec, L: 0
0113 08:54:35.497522 (+ 118us) write_op.cc:270] Start()
0113 08:54:35.497540 (+ 18us) write_op.cc:276] Timestamp: P: 1736758474286871 usec, L: 0
0113 08:54:35.497690 (+ 150us) write_op.cc:270] Start()
0113 08:54:35.497710 (+ 20us) write_op.cc:276] Timestamp: P: 1736758474310454 usec, L: 0
0113 08:54:35.497838 (+ 128us) write_op.cc:270] Start()
0113 08:54:35.497857 (+ 19us) write_op.cc:276] Timestamp: P: 1736758474349217 usec, L: 0
0113 08:54:35.497970 (+ 113us) write_op.cc:270] Start()
0113 08:54:35.497988 (+ 18us) write_op.cc:276] Timestamp: P: 1736758474372993 usec, L: 0
0113 08:54:35.498114 (+ 126us) write_op.cc:270] Start()
0113 08:54:35.498131 (+ 17us) write_op.cc:276] Timestamp: P: 1736758474373630 usec, L: 0
0113 08:54:35.568468 (+ 70337us) log.cc:844] Serialized 118764 byte log entry
0113 08:54:35.568727 (+ 259us) raft_consensus.cc:1640] Marking committed up to 915
0113 08:54:35.606364 (+ 37637us) raft_consensus.cc:1692] Filling consensus response to leader.
0113 08:54:35.606402 (+ 38us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0113 08:54:35.606412 (+ 10us) raft_consensus.cc:1680] finished
0113 08:54:35.606414 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0113 08:54:35.606509 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:54:35.370075 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 34cc82c767a44ffab6786f89b8d41a00
0113 08:54:35.370186 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:54:35.370192 (+ 6us) write_op.cc:435] Acquired schema lock
0113 08:54:35.370195 (+ 3us) tablet.cc:659] Decoding operations
0113 08:54:35.370386 (+ 191us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:54:35.370400 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0113 08:54:35.370405 (+ 5us) tablet.cc:682] Acquiring locks for 13 operations
0113 08:54:35.370681 (+ 276us) tablet.cc:698] Row locks acquired
0113 08:54:35.370684 (+ 3us) write_op.cc:260] PREPARE: finished
0113 08:54:35.568787 (+198103us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0113 08:54:35.370762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 34cc82c767a44ffab6786f89b8d41a00
0113 08:54:35.370854 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:54:35.370858 (+ 4us) write_op.cc:435] Acquired schema lock
0113 08:54:35.370860 (+ 2us) tablet.cc:659] Decoding operations
0113 08:54:35.370943 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:54:35.370954 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0113 08:54:35.370958 (+ 4us) tablet.cc:682] Acquiring locks for 6 operations
0113 08:54:35.371077 (+ 119us) tablet.cc:698] Row locks acquired
0113 08:54:35.371079 (+ 2us) write_op.cc:260] PREPARE: finished
0113 08:54:35.568845 (+197766us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0113 08:54:35.371175 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 34cc82c767a44ffab6786f89b8d41a00
0113 08:54:35.371251 (+ 76us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:54:35.371256 (+ 5us) write_op.cc:435] Acquired schema lock
0113 08:54:35.371259 (+ 3us) tablet.cc:659] Decoding operations
0113 08:54:35.389701 (+ 18442us) write_op.cc:620] Acquiring the partition loc
I20250113 08:54:36.075717 8586 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=204 overwritten=0 applied=202 ignored=0} inserts{seen=1650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 08:54:36.088482 8586 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap complete.
I20250113 08:54:36.100849 8586 ts_tablet_manager.cc:1397] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 5.906s user 0.467s sys 0.040s
I20250113 08:54:36.108795 8586 raft_consensus.cc:357] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:36.122511 8752 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:36.125586 8752 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } }
I20250113 08:54:36.126523 8752 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:54:36.132027 8586 raft_consensus.cc:738] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:36.134297 8586 consensus_queue.cc:260] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 202, Last appended: 1.204, Last appended by leader: 204, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:36.135071 8586 ts_tablet_manager.cc:1428] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20250113 08:54:36.135351 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 918 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:36.135493 8586 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
I20250113 08:54:36.136032 7967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cc82c767a44ffab6786f89b8d41a00" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 918 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:36.139230 8581 leader_election.cc:304] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:33.010257 8659 rpcz_store.cc:276] 0113 08:54:30.544928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:30.546851 (+ 1923us) service_pool.cc:225] Handling call
0113 08:54:30.697027 (+150176us) raft_consensus.cc:1466] Updating replica for 541 ops
0113 08:54:30.770876 (+ 73849us) raft_consensus.cc:1529] Early marking committed up to index 202
0113 08:54:30.852377 (+ 81501us) raft_consensus.cc:1534] Triggering prepare for 541 ops
0113 08:54:30.853037 (+ 660us) write_op.cc:270] Start()
0113 08:54:30.853066 (+ 29us) write_op.cc:276] Timestamp: P: 1736758458190949 usec, L: 0
0113 08:54:30.858554 (+ 5488us) write_op.cc:270] Start()
0113 08:54:30.858638 (+ 84us) write_op.cc:276] Timestamp: P: 1736758458196297 usec, L: 0
0113 08:54:30.915363 (+ 56725us) write_op.cc:270] Start()
0113 08:54:30.915398 (+ 35us) write_op.cc:276] Timestamp: P: 1736758458467404 usec, L: 0
0113 08:54:30.915569 (+ 171us) write_op.cc:270] Start()
0113 08:54:30.915595 (+ 26us) write_op.cc:276] Timestamp: P: 1736758458809990 usec, L: 0
0113 08:54:30.915741 (+ 146us) write_op.cc:270] Start()
0113 08:54:30.915759 (+ 18us) write_op.cc:276] Timestamp: P: 1736758458837023 usec, L: 0
0113 08:54:30.917729 (+ 1970us) write_op.cc:270] Start()
0113 08:54:30.917765 (+ 36us) write_op.cc:276] Timestamp: P: 1736758458853101 usec, L: 0
0113 08:54:30.917970 (+ 205us) write_op.cc:270] Start()
0113 08:54:30.918016 (+ 46us) write_op.cc:276] Timestamp: P: 1736758458859711 usec, L: 0
0113 08:54:30.918161 (+ 145us) write_op.cc:270] Start()
0113 08:54:30.918179 (+ 18us) write_op.cc:276] Timestamp: P: 1736758458892689 usec, L: 0
0113 08:54:30.918307 (+ 128us) write_op.cc:270] Start()
0113 08:54:30.918325 (+ 18us) write_op.cc:276] Timestamp: P: 1736758458909962 usec, L: 0
0113 08:54:30.918479 (+ 154us) write_op.cc:270] Start()
0113 08:54:30.918497 (+ 18us) write_op.cc:276] Timestamp: P: 1736758458952887 usec, L: 0
0113 08:54:30.918643 (+ 146us) write_op.cc:270] Start()
0113 08:54:30.918662 (+ 19us) write_op.cc:276] Timestamp: P: 1736758458955518 usec, L: 0
0113 08:54:30.918841 (+ 179us) write_op.cc:270] Start()
0113 08:54:30.918861 (+ 20us) write_op.cc:276] Timestamp: P: 1736758458976799 usec, L: 0
0113 08:54:30.919019 (+ 158us) write_op.cc:270] Start()
0113 08:54:30.919040 (+ 21us) write_op.cc:276] Timestamp: P: 1736758458995448 usec, L: 0
0113 08:54:30.919190 (+ 150us) write_op.cc:270] Start()
0113 08:54:30.919215 (+ 25us) write_op.cc:276] Timestamp: P: 1736758458998367 usec, L: 0
0113 08:54:30.919370 (+ 155us) write_op.cc:270] Start()
0113 08:54:30.919392 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459002536 usec, L: 0
0113 08:54:30.919548 (+ 156us) write_op.cc:270] Start()
0113 08:54:30.919569 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459043552 usec, L: 0
0113 08:54:30.919753 (+ 184us) write_op.cc:270] Start()
0113 08:54:30.919785 (+ 32us) write_op.cc:276] Timestamp: P: 1736758459047868 usec, L: 0
0113 08:54:30.919952 (+ 167us) write_op.cc:270] Start()
0113 08:54:30.919980 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459051949 usec, L: 0
0113 08:54:30.920170 (+ 190us) write_op.cc:270] Start()
0113 08:54:30.920193 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459057869 usec, L: 0
0113 08:54:30.920336 (+ 143us) write_op.cc:270] Start()
0113 08:54:30.920358 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459092860 usec, L: 0
0113 08:54:30.920506 (+ 148us) write_op.cc:270] Start()
0113 08:54:30.920526 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459108181 usec, L: 0
0113 08:54:30.920667 (+ 141us) write_op.cc:270] Start()
0113 08:54:30.920685 (+ 18us) write_op.cc:276] Timestamp: P: 1736758459113782 usec, L: 0
0113 08:54:30.920837 (+ 152us) write_op.cc:270] Start()
0113 08:54:30.920855 (+ 18us) write_op.cc:276] Timestamp: P: 1736758459115479 usec, L: 0
0113 08:54:30.920973 (+ 118us) write_op.cc:270] Start()
0113 08:54:30.920992 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459134161 usec, L: 0
0113 08:54:30.921131 (+ 139us) write_op.cc:270] Start()
0113 08:54:30.921149 (+ 18us) write_op.cc:276] Timestamp: P: 1736758459166690 usec, L: 0
0113 08:54:30.922536 (+ 1387us) write_op.cc:270] Start()
0113 08:54:30.922563 (+ 27us) write_op.cc:276] Timestamp: P: 1736758459183066 usec, L: 0
0113 08:54:30.922749 (+ 186us) write_op.cc:270] Start()
0113 08:54:30.922768 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459193198 usec, L: 0
0113 08:54:30.924348 (+ 1580us) write_op.cc:270] Start()
0113 08:54:30.924365 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459195159 usec, L: 0
0113 08:54:30.926084 (+ 1719us) write_op.cc:270] Start()
0113 08:54:30.926112 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459232366 usec, L: 0
0113 08:54:30.926263 (+ 151us) write_op.cc:270] Start()
0113 08:54:30.926280 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459234412 usec, L: 0
0113 08:54:30.926411 (+ 131us) write_op.cc:270] Start()
0113 08:54:30.926428 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459260970 usec, L: 0
0113 08:54:30.926566 (+ 138us) write_op.cc:270] Start()
0113 08:54:30.926585 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459261634 usec, L: 0
0113 08:54:30.937531 (+ 10946us) write_op.cc:270] Start()
0113 08:54:30.937555 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459290203 usec, L: 0
0113 08:54:30.938241 (+ 686us) write_op.cc:270] Start()
0113 08:54:30.938266 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459294213 usec, L: 0
0113 08:54:30.953792 (+ 15526us) write_op.cc:270] Start()
0113 08:54:30.953820 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459303769 usec, L: 0
0113 08:54:30.954004 (+ 184us) write_op.cc:270] Start()
0113 08:54:30.954024 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459313570 usec, L: 0
0113 08:54:30.968039 (+ 14015us) write_op.cc:270] Start()
0113 08:54:30.968096 (+ 57us) write_op.cc:276] Timestamp: P: 1736758459344893 usec, L: 0
0113 08:54:30.971464 (+ 3368us) write_op.cc:270] Start()
0113 08:54:30.971497 (+ 33us) write_op.cc:276] Timestamp: P: 1736758459359527 usec, L: 0
0113 08:54:30.974327 (+ 2830us) write_op.cc:270] Start()
0113 08:54:30.975748 (+ 1421us) write_op.cc:276] Timestamp: P: 1736758459361570 usec, L: 0
0113 08:54:30.983044 (+ 7296us) write_op.cc:270] Start()
0113 08:54:30.983107 (+ 63us) write_op.cc:276] Timestamp: P: 1736758459386871 usec, L: 0
0113 08:54:30.983409 (+ 302us) write_op.cc:270] Start()
0113 08:54:30.983431 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459412828 usec, L: 0
0113 08:54:30.989898 (+ 6467us) write_op.cc:270] Start()
0113 08:54:30.989933 (+ 35us) write_op.cc:276] Timestamp: P: 1736758459416416 usec, L: 0
0113 08:54:30.996468 (+ 6535us) write_op.cc:270] Start()
0113 08:54:30.996520 (+ 52us) write_op.cc:276] Timestamp: P: 1736758459422828 usec, L: 0
0113 08:54:31.000585 (+ 4065us) write_op.cc:270] Start()
0113 08:54:31.000625 (+ 40us) write_op.cc:276] Timestamp: P: 1736758459476752 usec, L: 0
0113 08:54:31.012657 (+ 12032us) write_op.cc:270] Start()
0113 08:54:31.012730 (+ 73us) write_op.cc:276] Timestamp: P: 1736758459491719 usec, L: 0
0113 08:54:31.017798 (+ 5068us) write_op.cc:270] Start()
0113 08:54:31.017833 (+ 35us) write_op.cc:276] Timestamp: P: 1736758459495876 usec, L: 0
0113 08:54:31.023939 (+ 6106us) write_op.cc:270] Start()
0113 08:54:31.023983 (+ 44us) write_op.cc:276] Timestamp: P: 1736758459512730 usec, L: 0
0113 08:54:31.027417 (+ 3434us) write_op.cc:270] Start()
0113 08:54:31.027440 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459515485 usec, L: 0
0113 08:54:31.031717 (+ 4277us) write_op.cc:270] Start()
0113 08:54:31.031759 (+ 42us) write_op.cc:276] Timestamp: P: 1736758459543709 usec, L: 0
0113 08:54:31.043469 (+ 11710us) write_op.cc:270] Start()
0113 08:54:31.043496 (+ 27us) write_op.cc:276] Timestamp: P: 1736758459547173 usec, L: 0
0113 08:54:31.056194 (+ 12698us) write_op.cc:270] Start()
0113 08:54:31.056247 (+ 53us) write_op.cc:276] Timestamp: P: 1736758459590244 usec, L: 0
0113 08:54:31.057288 (+ 1041us) write_op.cc:270] Start()
0113 08:54:31.057316 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459591627 usec, L: 0
0113 08:54:31.058577 (+ 1261us) write_op.cc:270] Start()
0113 08:54:31.058601 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459594934 usec, L: 0
0113 08:54:31.068411 (+ 9810us) write_op.cc:270] Start()
0113 08:54:31.068456 (+ 45us) write_op.cc:276] Timestamp: P: 1736758459607664 usec, L: 0
0113 08:54:31.076230 (+ 7774us) write_op.cc:270] Start()
0113 08:54:31.076271 (+ 41us) write_op.cc:276] Timestamp: P: 1736758459639653 usec, L: 0
0113 08:54:31.077864 (+ 1593us) write_op.cc:270] Start()
0113 08:54:31.077902 (+ 38us) write_op.cc:276] Timestamp: P: 1736758459644135 usec, L: 0
0113 08:54:31.083437 (+ 5535us) write_op.cc:270] Start()
0113 08:54:31.083473 (+ 36us) write_op.cc:276] Timestamp: P: 1736758459661659 usec, L: 0
0113 08:54:31.083707 (+ 234us) write_op.cc:270] Start()
0113 08:54:31.083731 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459666782 usec, L: 0
0113 08:54:31.087361 (+ 3630us) write_op.cc:270] Start()
0113 08:54:31.087397 (+ 36us) write_op.cc:276] Timestamp: P: 1736758459697578 usec, L: 0
0113 08:54:31.087622 (+ 225us) write_op.cc:270] Start()
0113 08:54:31.087708 (+ 86us) write_op.cc:276] Timestamp: P: 1736758459736068 usec, L: 0
0113 08:54:31.087889 (+ 181us) write_op.cc:270] Start()
0113 08:54:31.087917 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459742841 usec, L: 0
0113 08:54:31.088088 (+ 171us) write_op.cc:270] Start()
0113 08:54:31.088111 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459744941 usec, L: 0
0113 08:54:31.088251 (+ 140us) write_op.cc:270] Start()
0113 08:54:31.088271 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459745786 usec, L: 0
0113 08:54:31.088439 (+ 168us) write_op.cc:270] Start()
0113 08:54:31.088462 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459791784 usec, L: 0
0113 08:54:31.094963 (+ 6501us) write_op.cc:270] Start()
0113 08:54:31.095011 (+ 48us) write_op.cc:276] Timestamp: P: 1736758459792462 usec, L: 0
0113 08:54:31.095281 (+ 270us) write_op.cc:270] Start()
0113 08:54:31.095305 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459797017 usec, L: 0
0113 08:54:31.095504 (+ 199us) write_op.cc:270] Start()
0113 08:54:31.095530 (+ 26us) write_op.cc:276] Timestamp: P: 1736758459806394 usec, L: 0
0113 08:54:31.095821 (+ 291us) write_op.cc:270] Start()
0113 08:54:31.095853 (+ 32us) write_op.cc:276] Timestamp: P: 1736758459839302 usec, L: 0
0113 08:54:31.103796 (+ 7943us) write_op.cc:270] Start()
0113 08:54:31.103849 (+ 53us) write_op.cc:276] Timestamp: P: 1736758459841883 usec, L: 0
0113 08:54:31.104148 (+ 299us) write_op.cc:270] Start()
0113 08:54:31.104175 (+ 27us) write_op.cc:276] Timestamp: P: 1736758459864912 usec, L: 0
0113 08:54:31.104502 (+ 327us) write_op.cc:270] Start()
0113 08:54:31.104531 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459871756 usec, L: 0
0113 08:54:31.104775 (+ 244us) write_op.cc:270] Start()
0113 08:54:31.104806 (+ 31us) write_op.cc:276] Timestamp: P: 1736758459913359 usec, L: 0
0113 08:54:31.105139 (+ 333us) write_op.cc:270] Start()
0113 08:54:31.105183 (+ 44us) write_op.cc:276] Timestamp: P: 1736758459915775 usec, L: 0
0113 08:54:31.105343 (+ 160us) write_op.cc:270] Start()
0113 08:54:31.105363 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459934650 usec, L: 0
0113 08:54:31.105532 (+ 169us) write_op.cc:270] Start()
0113 08:54:31.105553 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459940620 usec, L: 0
0113 08:54:31.105750 (+ 197us) write_op.cc:270] Start()
0113 08:54:31.105775 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459985928 usec, L: 0
0113 08:54:31.115125 (+ 9350us) write_op.cc:270] Start()
0113 08:54:31.115178 (+ 53us) write_op.cc:276] Timestamp: P: 1736758459988821 usec, L: 0
0113 08:54:31.122636 (+ 7458us) write_op.cc:270] Start()
0113 08:54:31.122675 (+ 39us) write_op.cc:276] Timestamp: P: 1736758459993076 usec, L: 0
0113 08:54:31.122910 (+ 235us) write_op.cc:270] Start()
0113 08:54:31.123073 (+ 163us) write_op.cc:276] Timestamp: P: 1736758460014963 usec, L: 0
0113 08:54:31.130131 (+ 7058us) write_op.cc:270] Start()
0113 08:54:31.130169 (+ 38us) write_op.cc:276] Timestamp: P: 1736758460035104 usec, L: 0
0113 08:54:31.135604 (+ 5435us) write_op.cc:270] Start()
0113 08:54:31.135655 (+ 51us) write_op.cc:276] Timestamp: P: 1736758460040225 usec, L: 0
0113 08:54:31.161968 (+ 26313us) write_op.cc:270] Start()
0113 08:54:31.162006 (+ 38us) write_op.cc:276] Timestamp: P: 1736758460063055 usec, L: 0
0113 08:54:31.164212 (+ 2206us) write_op.cc:270] Start()
0113 08:54:31.164246 (+ 34us) write_op.cc:276] Timestamp: P: 1736758460066231 usec, L: 0
0113 08:54:31.164462 (+ 216us) write_op.cc:270] Start()
0113 08:54:31.164487 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460104688 usec, L: 0
0113 08:54:31.164668 (+ 181us) write_op.cc:270] Start()
0113 08:54:31.164690 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460106894 usec, L: 0
0113 08:54:31.164863 (+ 173us) write_op.cc:270] Start()
0113 08:54:31.164883 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460124104 usec, L: 0
0113 08:54:31.165045 (+ 162us) write_op.cc:270] Start()
0113 08:54:31.165065 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460130517 usec, L: 0
0113 08:54:31.165216 (+ 151us) write_op.cc:270] Start()
0113 08:54:31.165235 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460168500 usec, L: 0
0113 08:54:31.167216 (+ 1981us) write_op.cc:270] Start()
0113 08:54:31.167242 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460174303 usec, L: 0
0113 08:54:31.167424 (+ 182us) write_op.cc:270] Start()
0113 08:54:31.167448 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460186197 usec, L: 0
0113 08:54:31.167676 (+ 228us) write_op.cc:270] Start()
0113 08:54:31.167703 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460190501 usec, L: 0
0113 08:54:31.172405 (+ 4702us) write_op.cc:270] Start()
0113 08:54:31.189769 (+ 17364us) write_op.cc:276] Timestamp: P: 1736758460219557 usec, L: 0
0113 08:54:31.190410 (+ 641us) write_op.cc:270] Start()
0113 08:54:31.190437 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460231092 usec, L: 0
0113 08:54:31.192834 (+ 2397us) write_op.cc:270] Start()
0113 08:54:31.192858 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460252648 usec, L: 0
0113 08:54:31.194736 (+ 1878us) write_op.cc:270] Start()
0113 08:54:31.194755 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460254941 usec, L: 0
0113 08:54:31.205438 (+ 10683us) write_op.cc:270] Start()
0113 08:54:31.205473 (+ 35us) write_op.cc:276] Timestamp: P: 1736758460275914 usec, L: 0
0113 08:54:31.214042 (+ 8569us) write_op.cc:270] Start()
0113 08:54:31.214082 (+ 40us) write_op.cc:276] Timestamp: P: 1736758460288233 usec, L: 0
0113 08:54:31.214302 (+ 220us) write_op.cc:270] Start()
0113 08:54:31.214317 (+ 15us) write_op.cc:276] Timestamp: P: 1736758460300904 usec, L: 0
0113 08:54:31.218414 (+ 4097us) write_op.cc:270] Start()
0113 08:54:31.218470 (+ 56us) write_op.cc:276] Timestamp: P: 1736758460330734 usec, L: 0
0113 08:54:31.226104 (+ 7634us) write_op.cc:270] Start()
0113 08:54:31.226141 (+ 37us) write_op.cc:276] Timestamp: P: 1736758460341920 usec, L: 0
0113 08:54:31.229746 (+ 3605us) write_op.cc:270] Start()
0113 08:54:31.229778 (+ 32us) write_op.cc:276] Timestamp: P: 1736758460342654 usec, L: 0
0113 08:54:31.230852 (+ 1074us) write_op.cc:270] Start()
0113 08:54:31.230883 (+ 31us) write_op.cc:276] Timestamp: P: 1736758460363260 usec, L: 0
0113 08:54:31.231076 (+ 193us) write_op.cc:270] Start()
0113 08:54:31.231097 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460403791 usec, L: 0
0113 08:54:31.231799 (+ 702us) write_op.cc:270] Start()
0113 08:54:31.231826 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460419253 usec, L: 0
0113 08:54:31.232174 (+ 348us) write_op.cc:270] Start()
0113 08:54:31.232201 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460420906 usec, L: 0
0113 08:54:31.232366 (+ 165us) write_op.cc:270] Start()
0113 08:54:31.232387 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460432258 usec, L: 0
0113 08:54:31.232543 (+ 156us) write_op.cc:270] Start()
0113 08:54:31.232563 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460461589 usec, L: 0
0113 08:54:31.232702 (+ 139us) write_op.cc:270] Start()
0113 08:54:31.232758 (+ 56us) write_op.cc:276] Timestamp: P: 1736758460486285 usec, L: 0
0113 08:54:31.232936 (+ 178us) write_op.cc:270] Start()
0113 08:54:31.232958 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460486933 usec, L: 0
0113 08:54:31.233163 (+ 205us) write_op.cc:270] Start()
0113 08:54:31.233186 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460499696 usec, L: 0
0113 08:54:31.240487 (+ 7301us) write_op.cc:270] Start()
0113 08:54:31.240523 (+ 36us) write_op.cc:276] Timestamp: P: 1736758460520782 usec, L: 0
0113 08:54:31.240782 (+ 259us) write_op.cc:270] Start()
0113 08:54:31.240809 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460569726 usec, L: 0
0113 08:54:31.240986 (+ 177us) write_op.cc:270] Start()
0113 08:54:31.241008 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460571702 usec, L: 0
0113 08:54:31.241176 (+ 168us) write_op.cc:270] Start()
0113 08:54:31.241197 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460572529 usec, L: 0
0113 08:54:31.241373 (+ 176us) write_op.cc:270] Start()
0113 08:54:31.241395 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460600608 usec, L: 0
0113 08:54:31.241555 (+ 160us) write_op.cc:270] Start()
0113 08:54:31.241579 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460612434 usec, L: 0
0113 08:54:31.241731 (+ 152us) write_op.cc:270] Start()
0113 08:54:31.241752 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460645300 usec, L: 0
0113 08:54:31.241963 (+ 211us) write_op.cc:270] Start()
0113 08:54:31.241989 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460649771 usec, L: 0
0113 08:54:31.242172 (+ 183us) write_op.cc:270] Start()
0113 08:54:31.242195 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460654861 usec, L: 0
0113 08:54:31.242353 (+ 158us) write_op.cc:270] Start()
0113 08:54:31.242376 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460673662 usec, L: 0
0113 08:54:31.242552 (+ 176us) write_op.cc:270] Start()
0113 08:54:31.242574 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460710541 usec, L: 0
0113 08:54:31.242727 (+ 153us) write_op.cc:270] Start()
0113 08:54:31.242752 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460716772 usec, L: 0
0113 08:54:31.276072 (+ 33320us) write_op.cc:270] Start()
0113 08:54:31.276113 (+ 41us) write_op.cc:276] Timestamp: P: 1736758460732922 usec, L: 0
0113 08:54:31.276330 (+ 217us) write_op.cc:270] Start()
0113 08:54:31.276379 (+ 49us) write_op.cc:276] Timestamp: P: 1736758460735954 usec, L: 0
0113 08:54:31.276563 (+ 184us) write_op.cc:270] Start()
0113 08:54:31.276582 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460765657 usec, L: 0
0113 08:54:31.276724 (+ 142us) write_op.cc:270] Start()
0113 08:54:31.276741 (+ 17us) write_op.cc:276] Timestamp: P: 1736758460772711 usec, L: 0
0113 08:54:31.277774 (+ 1033us) write_op.cc:270] Start()
0113 08:54:31.277804 (+ 30us) write_op.cc:276] Timestamp: P: 1736758460773336 usec, L: 0
0113 08:54:31.287108 (+ 9304us) write_op.cc:270] Start()
0113 08:54:31.287155 (+ 47us) write_op.cc:276] Timestamp: P: 1736758460788594 usec, L: 0
0113 08:54:31.303795 (+ 16640us) write_op.cc:270] Start()
0113 08:54:31.303827 (+ 32us) write_op.cc:276] Timestamp: P: 1736758460830540 usec, L: 0
0113 08:54:31.313413 (+ 9586us) write_op.cc:270] Start()
0113 08:54:31.313449 (+ 36us) write_op.cc:276] Timestamp: P: 1736758460832052 usec, L: 0
0113 08:54:31.313681 (+ 232us) write_op.cc:270] Start()
0113 08:54:31.313734 (+ 53us) write_op.cc:276] Timestamp: P: 1736758460836601 usec, L: 0
0113 08:54:31.313961 (+ 227us) write_op.cc:270] Start()
0113 08:54:31.313987 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460865870 usec, L: 0
0113 08:54:31.314191 (+ 204us) write_op.cc:270] Start()
0113 08:54:31.314217 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460883676 usec, L: 0
0113 08:54:31.321229 (+ 7012us) write_op.cc:270] Start()
0113 08:54:31.321276 (+ 47us) write_op.cc:276] Timestamp: P: 1736758460885016 usec, L: 0
0113 08:54:31.321537 (+ 261us) write_op.cc:270] Start()
0113 08:54:31.321564 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460910533 usec, L: 0
0113 08:54:31.321738 (+ 174us) write_op.cc:270] Start()
0113 08:54:31.321755 (+ 17us) write_op.cc:276] Timestamp: P: 1736758460927905 usec, L: 0
0113 08:54:31.321853 (+ 98us) write_op.cc:270] Start()
0113 08:54:31.321866 (+ 13us) write_op.cc:276] Timestamp: P: 1736758460948521 usec, L: 0
0113 08:54:31.321979 (+ 113us) write_op.cc:270] Start()
0113 08:54:31.321992 (+ 13us) write_op.cc:276] Timestamp: P: 1736758460968350 usec, L: 0
0113 08:54:31.322082 (+ 90us) write_op.cc:270] Start()
0113 08:54:31.322095 (+ 13us) write_op.cc:276] Timestamp: P: 1736758460970644 usec, L: 0
0113 08:54:31.322799 (+ 704us) write_op.cc:270] Start()
0113 08:54:31.323840 (+ 1041us) write_op.cc:276] Timestamp: P: 1736758461010122 usec, L: 0
0113 08:54:31.324154 (+ 314us) write_op.cc:270] Start()
0113 08:54:31.324177 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461027013 usec, L: 0
0113 08:54:31.324337 (+ 160us) write_op.cc:270] Start()
0113 08:54:31.324358 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461029561 usec, L: 0
0113 08:54:31.324513 (+ 155us) write_op.cc:270] Start()
0113 08:54:31.324612 (+ 99us) write_op.cc:276] Timestamp: P: 1736758461034350 usec, L: 0
0113 08:54:31.324796 (+ 184us) write_op.cc:270] Start()
0113 08:54:31.324818 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461062461 usec, L: 0
0113 08:54:31.324977 (+ 159us) write_op.cc:270] Start()
0113 08:54:31.324998 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461066875 usec, L: 0
0113 08:54:31.328979 (+ 3981us) write_op.cc:270] Start()
0113 08:54:31.329013 (+ 34us) write_op.cc:276] Timestamp: P: 1736758461075168 usec, L: 0
0113 08:54:31.329264 (+ 251us) write_op.cc:270] Start()
0113 08:54:31.329286 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461093775 usec, L: 0
0113 08:54:31.329480 (+ 194us) write_op.cc:270] Start()
0113 08:54:31.329504 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461119362 usec, L: 0
0113 08:54:31.329654 (+ 150us) write_op.cc:270] Start()
0113 08:54:31.329675 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461124504 usec, L: 0
0113 08:54:31.329826 (+ 151us) write_op.cc:270] Start()
0113 08:54:31.329848 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461174620 usec, L: 0
0113 08:54:31.329989 (+ 141us) write_op.cc:270] Start()
0113 08:54:31.330008 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461175689 usec, L: 0
0113 08:54:31.350071 (+ 20063us) write_op.cc:270] Start()
0113 08:54:31.350096 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461184156 usec, L: 0
0113 08:54:31.351809 (+ 1713us) write_op.cc:270] Start()
0113 08:54:31.351840 (+ 31us) write_op.cc:276] Timestamp: P: 1736758461185362 usec, L: 0
0113 08:54:31.352063 (+ 223us) write_op.cc:270] Start()
0113 08:54:31.352088 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461221909 usec, L: 0
0113 08:54:31.352246 (+ 158us) write_op.cc:270] Start()
0113 08:54:31.352268 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461223318 usec, L: 0
0113 08:54:31.352440 (+ 172us) write_op.cc:270] Start()
0113 08:54:31.353050 (+ 610us) write_op.cc:276] Timestamp: P: 1736758461224011 usec, L: 0
0113 08:54:31.353218 (+ 168us) write_op.cc:270] Start()
0113 08:54:31.353240 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461225675 usec, L: 0
0113 08:54:31.353380 (+ 140us) write_op.cc:270] Start()
0113 08:54:31.353401 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461274087 usec, L: 0
0113 08:54:31.355134 (+ 1733us) write_op.cc:270] Start()
0113 08:54:31.355168 (+ 34us) write_op.cc:276] Timestamp: P: 1736758461293171 usec, L: 0
0113 08:54:31.357912 (+ 2744us) write_op.cc:270] Start()
0113 08:54:31.357934 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461294988 usec, L: 0
0113 08:54:31.359770 (+ 1836us) write_op.cc:270] Start()
0113 08:54:31.359798 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461298357 usec, L: 0
0113 08:54:31.359978 (+ 180us) write_op.cc:270] Start()
0113 08:54:31.359998 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461344756 usec, L: 0
0113 08:54:31.361773 (+ 1775us) write_op.cc:270] Start()
0113 08:54:31.361799 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461347189 usec, L: 0
0113 08:54:31.365099 (+ 3300us) write_op.cc:270] Start()
0113 08:54:31.365126 (+ 27us) write_op.cc:276] Timestamp: P: 1736758461347968 usec, L: 0
0113 08:54:31.369255 (+ 4129us) write_op.cc:270] Start()
0113 08:54:31.369288 (+ 33us) write_op.cc:276] Timestamp: P: 1736758461375536 usec, L: 0
0113 08:54:31.374107 (+ 4819us) write_op.cc:270] Start()
0113 08:54:31.374136 (+ 29us) write_op.cc:276] Timestamp: P: 1736758461400402 usec, L: 0
0113 08:54:31.379810 (+ 5674us) write_op.cc:270] Start()
0113 08:54:31.391496 (+ 11686us) write_op.cc:276] Timestamp: P: 1736758461405162 usec, L: 0
0113 08:54:31.399504 (+ 8008us) write_op.cc:270] Start()
0113 08:54:31.399520 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461408433 usec, L: 0
0113 08:54:31.405542 (+ 6022us) write_op.cc:270] Start()
0113 08:54:31.405571 (+ 29us) write_op.cc:276] Timestamp: P: 1736758461437457 usec, L: 0
0113 08:54:31.410838 (+ 5267us) write_op.cc:270] Start()
0113 08:54:31.410869 (+ 31us) write_op.cc:276] Timestamp: P: 1736758461474379 usec, L: 0
0113 08:54:31.423533 (+ 12664us) write_op.cc:270] Start()
0113 08:54:31.423686 (+ 153us) write_op.cc:276] Timestamp: P: 1736758461493019 usec, L: 0
0113 08:54:31.458421 (+ 34735us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x60f000016ef8. stack: 00007f4647092fd6 00007f4647092d90 00007f464607248d 000000000096d544 000000000096d50d 000000000096ceda 00007f464fdf4848 00007f4652974629 00007f465291a2ea 00007f4652917b47 00007f46566220cd 00007f4650cf5b56 00007f4650cf5885 00007f464ddfa30e 00007f464ddf91ee 00007f464ddff425
0113 08:54:31.458678 (+ 257us) write_op.cc:270] Start()
0113 08:54:31.458719 (+ 41us) write_op.cc:276] Timestamp: P: 1736758461495396 usec, L: 0
0113 08:54:31.458985 (+ 266us) write_op.cc:270] Start()
0113 08:54:31.459011 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461496245 usec, L: 0
0113 08:54:31.459182 (+ 171us) write_op.cc:270] Start()
0113 08:54:31.459204 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461545051 usec, L: 0
0113 08:54:31.459343 (+ 139us) write_op.cc:270] Start()
0113 08:54:31.459363 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461546184 usec, L: 0
0113 08:54:31.459543 (+ 180us) write_op.cc:270] Start()
0113 08:54:31.459566 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461547578 usec, L: 0
0113 08:54:31.459748 (+ 182us) write_op.cc:270] Start()
0113 08:54:31.459769 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461578347 usec, L: 0
0113 08:54:31.459921 (+ 152us) write_op.cc:270] Start()
0113 08:54:31.459950 (+ 29us) write_op.cc:276] Timestamp: P: 1736758461602302 usec, L: 0
0113 08:54:31.460132 (+ 182us) write_op.cc:270] Start()
0113 08:54:31.460155 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461643322 usec, L: 0
0113 08:54:31.460322 (+ 167us) write_op.cc:270] Start()
0113 08:54:31.460344 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461645333 usec, L: 0
0113 08:54:31.460495 (+ 151us) write_op.cc:270] Start()
0113 08:54:31.460515 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461655558 usec, L: 0
0113 08:54:31.474903 (+ 14388us) write_op.cc:270] Start()
0113 08:54:31.474941 (+ 38us) write_op.cc:276] Timestamp: P: 1736758461672660 usec, L: 0
0113 08:54:31.475195 (+ 254us) write_op.cc:270] Start()
0113 08:54:31.475220 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461705968 usec, L: 0
0113 08:54:31.475378 (+ 158us) write_op.cc:270] Start()
0113 08:54:31.475401 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461710457 usec, L: 0
0113 08:54:31.475574 (+ 173us) write_op.cc:270] Start()
0113 08:54:31.475594 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461727139 usec, L: 0
0113 08:54:31.475748 (+ 154us) write_op.cc:270] Start()
0113 08:54:31.475772 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461745707 usec, L: 0
0113 08:54:31.475940 (+ 168us) write_op.cc:270] Start()
0113 08:54:31.475968 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461769242 usec, L: 0
0113 08:54:31.476206 (+ 238us) write_op.cc:270] Start()
0113 08:54:31.476242 (+ 36us) write_op.cc:276] Timestamp: P: 1736758461781398 usec, L: 0
0113 08:54:31.476467 (+ 225us) write_op.cc:270] Start()
0113 08:54:31.476498 (+ 31us) write_op.cc:276] Timestamp: P: 1736758461787765 usec, L: 0
0113 08:54:31.476752 (+ 254us) write_op.cc:270] Start()
0113 08:54:31.476782 (+ 30us) write_op.cc:276] Timestamp: P: 1736758461829714 usec, L: 0
0113 08:54:31.476960 (+ 178us) write_op.cc:270] Start()
0113 08:54:31.476987 (+ 27us) write_op.cc:276] Timestamp: P: 1736758461855186 usec, L: 0
0113 08:54:31.477167 (+ 180us) write_op.cc:270] Start()
0113 08:54:31.477191 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461891299 usec, L: 0
0113 08:54:31.477352 (+ 161us) write_op.cc:270] Start()
0113 08:54:31.477377 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461908894 usec, L: 0
0113 08:54:31.477508 (+ 131us) write_op.cc:270] Start()
0113 08:54:31.477529 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461910402 usec, L: 0
0113 08:54:31.477706 (+ 177us) write_op.cc:270]
W20250113 08:54:36.156729 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20250113 08:54:36.679302 8752 raft_consensus.cc:2743] T 34cc82c767a44ffab6786f89b8d41a00 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:36.681665 8657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.3:55187 (request call id 16) took 1455 ms. Trace:
I20250113 08:54:36.673943 8754 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:36.694494 8754 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:36.718849 8754 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:36.719079 8660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:37056 (request call id 37) took 1056 ms. Trace:
I20250113 08:54:36.719192 8660 rpcz_store.cc:276] 0113 08:54:35.656485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:35.656557 (+ 72us) service_pool.cc:225] Handling call
0113 08:54:36.704491 (+1047934us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x616000019f54. stack: 00007f4647092fd6 00007f4647092d90 00007f464607248d 000000000096d544 000000000096d50d 00007f4655c30e1d 00007f4655c30f13 00007f4652939211 00007f4656630c3e 00007f4650cfdcb6 00007f4650cfd9e5 00007f464ddfa30e 00007f464ddf91ee 00007f464ddff425 00007f464ddffd4f 00007f464ddffb78
0113 08:54:36.713281 (+ 8790us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2402231680}
I20250113 08:54:36.722764 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 951 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:36.730624 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 951 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:36.733700 8581 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:36.737012 8754 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:36.802760 8662 raft_consensus.cc:3054] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:54:36.860666 8662 pending_rounds.cc:77] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Aborting all ops after (but not including) 202
I20250113 08:54:36.862753 8662 pending_rounds.cc:99] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Aborting uncommitted WRITE_OP operation due to leader change: 1.203
I20250113 08:54:36.863251 8662 pending_rounds.cc:99] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1: Aborting uncommitted WRITE_OP operation due to leader change: 1.204
I20250113 08:54:37.121978 8754 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:37.124912 8754 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:37.126839 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 951 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:37.130947 8754 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:37.138533 7968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 951 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:37.144276 8583 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:37.165947 8754 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:36.696494 8657 rpcz_store.cc:276] 0113 08:54:35.216413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:35.216478 (+ 65us) service_pool.cc:225] Handling call
0113 08:54:35.324402 (+107924us) raft_consensus.cc:1466] Updating replica for 207 ops
0113 08:54:35.433969 (+109567us) raft_consensus.cc:1529] Early marking committed up to index 744
0113 08:54:35.434089 (+ 120us) raft_consensus.cc:1534] Triggering prepare for 207 ops
0113 08:54:35.434233 (+ 144us) write_op.cc:270] Start()
0113 08:54:35.434276 (+ 43us) write_op.cc:276] Timestamp: P: 1736758470339512 usec, L: 0
0113 08:54:35.434541 (+ 265us) write_op.cc:270] Start()
0113 08:54:35.434564 (+ 23us) write_op.cc:276] Timestamp: P: 1736758470398606 usec, L: 0
0113 08:54:35.435237 (+ 673us) write_op.cc:270] Start()
0113 08:54:35.435264 (+ 27us) write_op.cc:276] Timestamp: P: 1736758470400334 usec, L: 0
0113 08:54:35.435437 (+ 173us) write_op.cc:270] Start()
0113 08:54:35.435458 (+ 21us) write_op.cc:276] Timestamp: P: 1736758470405234 usec, L: 0
0113 08:54:35.435617 (+ 159us) write_op.cc:270] Start()
0113 08:54:35.435641 (+ 24us) write_op.cc:276] Timestamp: P: 1736758470454266 usec, L: 0
0113 08:54:35.435771 (+ 130us) write_op.cc:270] Start()
0113 08:54:35.435789 (+ 18us) write_op.cc:276] Timestamp: P: 1736758470482982 usec, L: 0
0113 08:54:35.435907 (+ 118us) write_op.cc:270] Start()
0113 08:54:35.435940 (+ 33us) write_op.cc:276] Timestamp: P: 1736758470517655 usec, L: 0
0113 08:54:35.436057 (+ 117us) write_op.cc:270] Start()
0113 08:54:35.436072 (+ 15us) write_op.cc:276] Timestamp: P: 1736758470520324 usec, L: 0
0113 08:54:35.436220 (+ 148us) write_op.cc:270] Start()
0113 08:54:35.436236 (+ 16us) write_op.cc:276] Timestamp: P: 1736758470545869 usec, L: 0
0113 08:54:35.436360 (+ 124us) write_op.cc:270] Start()
0113 08:54:35.436379 (+ 19us) write_op.cc:276] Timestamp: P: 1736758470584594 usec, L: 0
0113 08:54:35.436567 (+ 188us) write_op.cc:270] Start()
0113 08:54:35.436589 (+ 22us) write_op.cc:276] Timestamp: P: 1736758470603175 usec, L: 0
0113 08:54:35.494822 (+ 58233us) write_op.cc:270] Start()
0113 08:54:35.494852 (+ 30us) write_op.cc:276] Timestamp: P: 1736758470625654 usec, L: 0
0113 08:54:35.502845 (+ 7993us) write_op.cc:270] Start()
0113 08:54:35.502895 (+ 50us) write_op.cc:276] Timestamp: P: 1736758470632878 usec, L: 0
0113 08:54:35.508977 (+ 6082us) write_op.cc:270] Start()
0113 08:54:35.509012 (+ 35us) write_op.cc:276] Timestamp: P: 1736758470673320 usec, L: 0
0113 08:54:35.517383 (+ 8371us) write_op.cc:270] Start()
0113 08:54:35.517414 (+ 31us) write_op.cc:276] Timestamp: P: 1736758470693326 usec, L: 0
0113 08:54:35.535867 (+ 18453us) write_op.cc:270] Start()
0113 08:54:35.535905 (+ 38us) write_op.cc:276] Timestamp: P: 1736758470697670 usec, L: 0
0113 08:54:35.538544 (+ 2639us) write_op.cc:270] Start()
0113 08:54:35.538587 (+ 43us) write_op.cc:276] Timestamp: P: 1736758470753298 usec, L: 0
0113 08:54:35.540220 (+ 1633us) write_op.cc:270] Start()
0113 08:54:35.540252 (+ 32us) write_op.cc:276] Timestamp: P: 1736758470779303 usec, L: 0
0113 08:54:35.544972 (+ 4720us) write_op.cc:270] Start()
0113 08:54:35.545012 (+ 40us) write_op.cc:276] Timestamp: P: 1736758470824384 usec, L: 0
0113 08:54:35.545268 (+ 256us) write_op.cc:270] Start()
0113 08:54:35.545292 (+ 24us) write_op.cc:276] Timestamp: P: 1736758470844844 usec, L: 0
0113 08:54:35.547190 (+ 1898us) write_op.cc:270] Start()
0113 08:54:35.547219 (+ 29us) write_op.cc:276] Timestamp: P: 1736758470847866 usec, L: 0
0113 08:54:35.553590 (+ 6371us) write_op.cc:270] Start()
0113 08:54:35.553625 (+ 35us) write_op.cc:276] Timestamp: P: 1736758470904245 usec, L: 0
0113 08:54:35.564113 (+ 10488us) write_op.cc:270] Start()
0113 08:54:35.564140 (+ 27us) write_op.cc:276] Timestamp: P: 1736758470909822 usec, L: 0
0113 08:54:35.564357 (+ 217us) write_op.cc:270] Start()
0113 08:54:35.564374 (+ 17us) write_op.cc:276] Timestamp: P: 1736758470910530 usec, L: 0
0113 08:54:35.565788 (+ 1414us) write_op.cc:270] Start()
0113 08:54:35.565820 (+ 32us) write_op.cc:276] Timestamp: P: 1736758470912773 usec, L: 0
0113 08:54:35.569602 (+ 3782us) write_op.cc:270] Start()
0113 08:54:35.569633 (+ 31us) write_op.cc:276] Timestamp: P: 1736758471004864 usec, L: 0
0113 08:54:35.575053 (+ 5420us) write_op.cc:270] Start()
0113 08:54:35.575092 (+ 39us) write_op.cc:276] Timestamp: P: 1736758471029187 usec, L: 0
0113 08:54:35.575374 (+ 282us) write_op.cc:270] Start()
0113 08:54:35.575396 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471034980 usec, L: 0
0113 08:54:35.575589 (+ 193us) write_op.cc:270] Start()
0113 08:54:35.575638 (+ 49us) write_op.cc:276] Timestamp: P: 1736758471036179 usec, L: 0
0113 08:54:35.595239 (+ 19601us) write_op.cc:270] Start()
0113 08:54:35.595280 (+ 41us) write_op.cc:276] Timestamp: P: 1736758471072632 usec, L: 0
0113 08:54:35.595572 (+ 292us) write_op.cc:270] Start()
0113 08:54:35.595597 (+ 25us) write_op.cc:276] Timestamp: P: 1736758471120494 usec, L: 0
0113 08:54:35.603512 (+ 7915us) write_op.cc:270] Start()
0113 08:54:35.603544 (+ 32us) write_op.cc:276] Timestamp: P: 1736758471149115 usec, L: 0
0113 08:54:35.608229 (+ 4685us) write_op.cc:270] Start()
0113 08:54:35.608266 (+ 37us) write_op.cc:276] Timestamp: P: 1736758471150175 usec, L: 0
0113 08:54:35.625995 (+ 17729us) write_op.cc:270] Start()
0113 08:54:35.626033 (+ 38us) write_op.cc:276] Timestamp: P: 1736758471152767 usec, L: 0
0113 08:54:35.647614 (+ 21581us) write_op.cc:270] Start()
0113 08:54:35.647651 (+ 37us) write_op.cc:276] Timestamp: P: 1736758471180387 usec, L: 0
0113 08:54:35.647934 (+ 283us) write_op.cc:270] Start()
0113 08:54:35.647960 (+ 26us) write_op.cc:276] Timestamp: P: 1736758471237908 usec, L: 0
0113 08:54:35.648167 (+ 207us) write_op.cc:270] Start()
0113 08:54:35.648189 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471241231 usec, L: 0
0113 08:54:35.653215 (+ 5026us) write_op.cc:270] Start()
0113 08:54:35.653264 (+ 49us) write_op.cc:276] Timestamp: P: 1736758471245462 usec, L: 0
0113 08:54:35.654075 (+ 811us) write_op.cc:270] Start()
0113 08:54:35.654111 (+ 36us) write_op.cc:276] Timestamp: P: 1736758471255790 usec, L: 0
0113 08:54:35.657683 (+ 3572us) write_op.cc:270] Start()
0113 08:54:35.657840 (+ 157us) write_op.cc:276] Timestamp: P: 1736758471303282 usec, L: 0
0113 08:54:35.681337 (+ 23497us) write_op.cc:270] Start()
0113 08:54:35.681385 (+ 48us) write_op.cc:276] Timestamp: P: 1736758471312099 usec, L: 0
0113 08:54:35.733736 (+ 52351us) write_op.cc:270] Start()
0113 08:54:35.733761 (+ 25us) write_op.cc:276] Timestamp: P: 1736758471331635 usec, L: 0
0113 08:54:35.735696 (+ 1935us) write_op.cc:270] Start()
0113 08:54:35.735727 (+ 31us) write_op.cc:276] Timestamp: P: 1736758471332465 usec, L: 0
0113 08:54:35.735989 (+ 262us) write_op.cc:270] Start()
0113 08:54:35.736008 (+ 19us) write_op.cc:276] Timestamp: P: 1736758471403387 usec, L: 0
0113 08:54:35.736168 (+ 160us) write_op.cc:270] Start()
0113 08:54:35.736184 (+ 16us) write_op.cc:276] Timestamp: P: 1736758471404240 usec, L: 0
0113 08:54:35.736344 (+ 160us) write_op.cc:270] Start()
0113 08:54:35.736361 (+ 17us) write_op.cc:276] Timestamp: P: 1736758471423854 usec, L: 0
0113 08:54:35.736505 (+ 144us) write_op.cc:270] Start()
0113 08:54:35.736525 (+ 20us) write_op.cc:276] Timestamp: P: 1736758471488456 usec, L: 0
0113 08:54:35.736685 (+ 160us) write_op.cc:270] Start()
0113 08:54:35.736701 (+ 16us) write_op.cc:276] Timestamp: P: 1736758471510563 usec, L: 0
0113 08:54:35.736841 (+ 140us) write_op.cc:270] Start()
0113 08:54:35.736860 (+ 19us) write_op.cc:276] Timestamp: P: 1736758471545184 usec, L: 0
0113 08:54:35.737006 (+ 146us) write_op.cc:270] Start()
0113 08:54:35.737021 (+ 15us) write_op.cc:276] Timestamp: P: 1736758471560983 usec, L: 0
0113 08:54:35.769658 (+ 32637us) write_op.cc:270] Start()
0113 08:54:35.769689 (+ 31us) write_op.cc:276] Timestamp: P: 1736758471577400 usec, L: 0
0113 08:54:35.778669 (+ 8980us) write_op.cc:270] Start()
0113 08:54:35.778710 (+ 41us) write_op.cc:276] Timestamp: P: 1736758471579540 usec, L: 0
0113 08:54:35.779015 (+ 305us) write_op.cc:270] Start()
0113 08:54:35.779037 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471651271 usec, L: 0
0113 08:54:35.779235 (+ 198us) write_op.cc:270] Start()
0113 08:54:35.779259 (+ 24us) write_op.cc:276] Timestamp: P: 1736758471674926 usec, L: 0
0113 08:54:35.779464 (+ 205us) write_op.cc:270] Start()
0113 08:54:35.779486 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471676244 usec, L: 0
0113 08:54:35.779683 (+ 197us) write_op.cc:270] Start()
0113 08:54:35.779705 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471679740 usec, L: 0
0113 08:54:35.820803 (+ 41098us) write_op.cc:270] Start()
0113 08:54:35.820832 (+ 29us) write_op.cc:276] Timestamp: P: 1736758471701826 usec, L: 0
0113 08:54:35.832637 (+ 11805us) write_op.cc:270] Start()
0113 08:54:35.832676 (+ 39us) write_op.cc:276] Timestamp: P: 1736758471744570 usec, L: 0
0113 08:54:35.855786 (+ 23110us) write_op.cc:270] Start()
0113 08:54:35.855828 (+ 42us) write_op.cc:276] Timestamp: P: 1736758471752086 usec, L: 0
0113 08:54:35.856393 (+ 565us) write_op.cc:270] Start()
0113 08:54:35.856443 (+ 50us) write_op.cc:276] Timestamp: P: 1736758471753433 usec, L: 0
0113 08:54:35.857656 (+ 1213us) write_op.cc:270] Start()
0113 08:54:35.857686 (+ 30us) write_op.cc:276] Timestamp: P: 1736758471786782 usec, L: 0
0113 08:54:35.871537 (+ 13851us) write_op.cc:270] Start()
0113 08:54:35.871572 (+ 35us) write_op.cc:276] Timestamp: P: 1736758471826003 usec, L: 0
0113 08:54:35.880757 (+ 9185us) write_op.cc:270] Start()
0113 08:54:35.880798 (+ 41us) write_op.cc:276] Timestamp: P: 1736758471834407 usec, L: 0
0113 08:54:35.910990 (+ 30192us) write_op.cc:270] Start()
0113 08:54:35.911022 (+ 32us) write_op.cc:276] Timestamp: P: 1736758471851357 usec, L: 0
0113 08:54:35.911288 (+ 266us) write_op.cc:270] Start()
0113 08:54:35.911308 (+ 20us) write_op.cc:276] Timestamp: P: 1736758471931068 usec, L: 0
0113 08:54:35.911490 (+ 182us) write_op.cc:270] Start()
0113 08:54:35.911508 (+ 18us) write_op.cc:276] Timestamp: P: 1736758471942096 usec, L: 0
0113 08:54:35.911670 (+ 162us) write_op.cc:270] Start()
0113 08:54:35.911687 (+ 17us) write_op.cc:276] Timestamp: P: 1736758471962117 usec, L: 0
0113 08:54:35.911858 (+ 171us) write_op.cc:270] Start()
0113 08:54:35.911874 (+ 16us) write_op.cc:276] Timestamp: P: 1736758471962654 usec, L: 0
0113 08:54:35.912029 (+ 155us) write_op.cc:270] Start()
0113 08:54:35.912050 (+ 21us) write_op.cc:276] Timestamp: P: 1736758471993183 usec, L: 0
0113 08:54:35.912213 (+ 163us) write_op.cc:270] Start()
0113 08:54:35.912234 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472038680 usec, L: 0
0113 08:54:35.912403 (+ 169us) write_op.cc:270] Start()
0113 08:54:35.912426 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472044819 usec, L: 0
0113 08:54:35.912568 (+ 142us) write_op.cc:270] Start()
0113 08:54:35.912588 (+ 20us) write_op.cc:276] Timestamp: P: 1736758472045454 usec, L: 0
0113 08:54:35.912756 (+ 168us) write_op.cc:270] Start()
0113 08:54:35.912778 (+ 22us) write_op.cc:276] Timestamp: P: 1736758472064724 usec, L: 0
0113 08:54:35.915947 (+ 3169us) write_op.cc:270] Start()
0113 08:54:35.915976 (+ 29us) write_op.cc:276] Timestamp: P: 1736758472091492 usec, L: 0
0113 08:54:35.916219 (+ 243us) write_op.cc:270] Start()
0113 08:54:35.916242 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472155022 usec, L: 0
0113 08:54:35.918395 (+ 2153us) write_op.cc:270] Start()
0113 08:54:35.918427 (+ 32us) write_op.cc:276] Timestamp: P: 1736758472156566 usec, L: 0
0113 08:54:35.922688 (+ 4261us) write_op.cc:270] Start()
0113 08:54:35.922720 (+ 32us) write_op.cc:276] Timestamp: P: 1736758472168261 usec, L: 0
0113 08:54:35.922948 (+ 228us) write_op.cc:270] Start()
0113 08:54:35.922970 (+ 22us) write_op.cc:276] Timestamp: P: 1736758472187930 usec, L: 0
0113 08:54:35.923123 (+ 153us) write_op.cc:270] Start()
0113 08:54:35.923144 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472219592 usec, L: 0
0113 08:54:35.923286 (+ 142us) write_op.cc:270] Start()
0113 08:54:35.923307 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472244875 usec, L: 0
0113 08:54:35.923492 (+ 185us) write_op.cc:270] Start()
0113 08:54:35.923514 (+ 22us) write_op.cc:276] Timestamp: P: 1736758472248006 usec, L: 0
0113 08:54:35.923676 (+ 162us) write_op.cc:270] Start()
0113 08:54:35.923699 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472277797 usec, L: 0
0113 08:54:35.923857 (+ 158us) write_op.cc:270] Start()
0113 08:54:35.923877 (+ 20us) write_op.cc:276] Timestamp: P: 1736758472296386 usec, L: 0
0113 08:54:35.924060 (+ 183us) write_op.cc:270] Start()
0113 08:54:35.924083 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472302421 usec, L: 0
0113 08:54:35.924246 (+ 163us) write_op.cc:270] Start()
0113 08:54:35.924269 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472329714 usec, L: 0
0113 08:54:35.924494 (+ 225us) write_op.cc:270] Start()
0113 08:54:35.924516 (+ 22us) write_op.cc:276] Timestamp: P: 1736758472359973 usec, L: 0
0113 08:54:35.924677 (+ 161us) write_op.cc:270] Start()
0113 08:54:35.924697 (+ 20us) write_op.cc:276] Timestamp: P: 1736758472369187 usec, L: 0
0113 08:54:35.924835 (+ 138us) write_op.cc:270] Start()
0113 08:54:35.924857 (+ 22us) write_op.cc:276] Timestamp: P: 1736758472422417 usec, L: 0
0113 08:54:35.930925 (+ 6068us) write_op.cc:270] Start()
0113 08:54:35.930963 (+ 38us) write_op.cc:276] Timestamp: P: 1736758472424392 usec, L: 0
0113 08:54:35.975165 (+ 44202us) write_op.cc:270] Start()
0113 08:54:35.975206 (+ 41us) write_op.cc:276] Timestamp: P: 1736758472433432 usec, L: 0
0113 08:54:35.977647 (+ 2441us) write_op.cc:270] Start()
0113 08:54:35.977683 (+ 36us) write_op.cc:276] Timestamp: P: 1736758472435008 usec, L: 0
0113 08:54:35.977919 (+ 236us) write_op.cc:270] Start()
0113 08:54:35.977942 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472532490 usec, L: 0
0113 08:54:35.978117 (+ 175us) write_op.cc:270] Start()
0113 08:54:35.978142 (+ 25us) write_op.cc:276] Timestamp: P: 1736758472535356 usec, L: 0
0113 08:54:35.978321 (+ 179us) write_op.cc:270] Start()
0113 08:54:35.978342 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472541144 usec, L: 0
0113 08:54:35.978492 (+ 150us) write_op.cc:270] Start()
0113 08:54:35.978511 (+ 19us) write_op.cc:276] Timestamp: P: 1736758472547646 usec, L: 0
0113 08:54:35.980126 (+ 1615us) write_op.cc:270] Start()
0113 08:54:35.980168 (+ 42us) write_op.cc:276] Timestamp: P: 1736758472624768 usec, L: 0
0113 08:54:35.980403 (+ 235us) write_op.cc:270] Start()
0113 08:54:35.980424 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472627616 usec, L: 0
0113 08:54:35.980586 (+ 162us) write_op.cc:270] Start()
0113 08:54:35.980609 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472628769 usec, L: 0
0113 08:54:35.980812 (+ 203us) write_op.cc:270] Start()
0113 08:54:35.980834 (+ 22us) write_op.cc:276] Timestamp: P: 1736758472629432 usec, L: 0
0113 08:54:35.981000 (+ 166us) write_op.cc:270] Start()
0113 08:54:35.981023 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472709821 usec, L: 0
0113 08:54:35.981195 (+ 172us) write_op.cc:270] Start()
0113 08:54:35.981224 (+ 29us) write_op.cc:276] Timestamp: P: 1736758472717819 usec, L: 0
0113 08:54:35.988016 (+ 6792us) write_op.cc:270] Start()
0113 08:54:35.988069 (+ 53us) write_op.cc:276] Timestamp: P: 1736758472720117 usec, L: 0
0113 08:54:35.988375 (+ 306us) write_op.cc:270] Start()
0113 08:54:35.988400 (+ 25us) write_op.cc:276] Timestamp: P: 1736758472731081 usec, L: 0
0113 08:54:35.988572 (+ 172us) write_op.cc:270] Start()
0113 08:54:35.988589 (+ 17us) write_op.cc:276] Timestamp: P: 1736758472809157 usec, L: 0
0113 08:54:35.988777 (+ 188us) write_op.cc:270] Start()
0113 08:54:35.988794 (+ 17us) write_op.cc:276] Timestamp: P: 1736758472813989 usec, L: 0
0113 08:54:36.002938 (+ 14144us) write_op.cc:270] Start()
0113 08:54:36.002972 (+ 34us) write_op.cc:276] Timestamp: P: 1736758472830709 usec, L: 0
0113 08:54:36.011383 (+ 8411us) write_op.cc:270] Start()
0113 08:54:36.011417 (+ 34us) write_op.cc:276] Timestamp: P: 1736758472836250 usec, L: 0
0113 08:54:36.014242 (+ 2825us) write_op.cc:270] Start()
0113 08:54:36.014268 (+ 26us) write_op.cc:276] Timestamp: P: 1736758472934929 usec, L: 0
0113 08:54:36.016601 (+ 2333us) write_op.cc:270] Start()
0113 08:54:36.016642 (+ 41us) write_op.cc:276] Timestamp: P: 1736758472943556 usec, L: 0
0113 08:54:36.022750 (+ 6108us) write_op.cc:270] Start()
0113 08:54:36.022899 (+ 149us) write_op.cc:276] Timestamp: P: 1736758472956054 usec, L: 0
0113 08:54:36.023179 (+ 280us) write_op.cc:270] Start()
0113 08:54:36.023200 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472976629 usec, L: 0
0113 08:54:36.023349 (+ 149us) write_op.cc:270] Start()
0113 08:54:36.023368 (+ 19us) write_op.cc:276] Timestamp: P: 1736758472993189 usec, L: 0
0113 08:54:36.023519 (+ 151us) write_op.cc:270] Start()
0113 08:54:36.023537 (+ 18us) write_op.cc:276] Timestamp: P: 1736758473033080 usec, L: 0
0113 08:54:36.027789 (+ 4252us) write_op.cc:270] Start()
0113 08:54:36.027825 (+ 36us) write_op.cc:276] Timestamp: P: 1736758473042573 usec, L: 0
0113 08:54:36.028110 (+ 285us) write_op.cc:270] Start()
0113 08:54:36.028136 (+ 26us) write_op.cc:276] Timestamp: P: 1736758473048639 usec, L: 0
0113 08:54:36.028320 (+ 184us) write_op.cc:270] Start()
0113 08:54:36.028343 (+ 23us) write_op.cc:276] Timestamp: P: 1736758473073472 usec, L: 0
0113 08:54:36.028504 (+ 161us) write_op.cc:270] Start()
0113 08:54:36.028526 (+ 22us) write_op.cc:276] Timestamp: P: 1736758473083544 usec, L: 0
0113 08:54:36.028725 (+ 199us) write_op.cc:270] Start()
0113 08:54:36.028750 (+ 25us) write_op.cc:276] Timestamp: P: 1736758473124771 usec, L: 0
0113 08:54:36.028938 (+ 188us) write_op.cc:270] Start()
0113 08:54:36.028962 (+ 24us) write_op.cc:276] Timestamp: P: 1736758473127656 usec, L: 0
0113 08:54:36.029117 (+ 155us) write_op.cc:270] Start()
0113 08:54:36.029139 (+ 22us) write_op.cc:276] Timestamp: P: 1736758473170181 usec, L: 0
0113 08:54:36.029298 (+ 159us) write_op.cc:270] Start()
0113 08:54:36.029322 (+ 24us) write_op.cc:276] Timestamp: P: 1736758473192120 usec, L: 0
0113 08:54:36.029504 (+ 182us) write_op.cc:270] Start()
0113 08:54:36.029527 (+ 23us) write_op.cc:276] Timestamp: P: 1736758473227814 usec, L: 0
0113 08:54:36.029718 (+ 191us) write_op.cc:270] Start()
0113 08:54:36.029741 (+ 23us) write_op.cc:276] Timestamp: P: 1736758473230392 usec, L: 0
0113 08:54:36.029906 (+ 165us) write_op.cc:270] Start()
0113 08:54:36.029928 (+ 22us) write_op.cc:276] Timestamp: P: 1736758473231373 usec, L: 0
0113 08:54:36.030082 (+ 154us) write_op.cc:270] Start()
0113 08:54:36.030104 (+ 22us) write_op.cc:276] Timestamp: P: 1736758473248197 usec, L: 0
0113 08:54:36.030247 (+ 143us) write_op.cc:270] Start()
0113 08:54:36.030267 (+ 20us) write_op.cc:276] Timestamp: P: 1736758473279451 usec, L: 0
0113 08:54:36.044697 (+ 14430us) write_op.cc:270] Start()
0113 08:54:36.044750 (+ 53us) write_op.cc:276] Timestamp: P: 1736758473319483 usec, L: 0
0113 08:54:36.077085 (+ 32335us) write_op.cc:270] Start()
0113 08:54:36.077122 (+ 37us) write_op.cc:276] Timestamp: P: 1736758473325388 usec, L: 0
0113 08:54:36.084189 (+ 7067us) write_op.cc:270] Start()
0113 08:54:36.084227 (+ 38us) write_op.cc:276] Timestamp: P: 1736758473350390 usec, L: 0
0113 08:54:36.084480 (+ 253us) write_op.cc:270] Start()
0113 08:54:36.084503 (+ 23us) write_op.cc:276] Timestamp: P: 1736758473385439 usec, L: 0
0113 08:54:36.086729 (+ 2226us) write_op.cc:270] Start()
0113 08:54:36.091986 (+ 5257us) write_op.cc:276] Timestamp: P: 1736758473397009 usec, L: 0
0113 08:54:36.100155 (+ 8169us) write_op.cc:270] Start()
0113 08:54:36.100190 (+ 35us) write_op.cc:276] Timestamp: P: 1736758473411419 usec, L: 0
0113 08:54:36.102074 (+ 1884us) write_op.cc:270] Start()
0113 08:54:36.102107 (+ 33us) write_op.cc:276] Timestamp: P: 1736758473473768 usec, L: 0
0113 08:54:36.106791 (+ 4684us) write_op.cc:270] Start()
0113 08:54:36.106883 (+ 92us) write_op.cc:276] Timestamp: P: 1736758473476706 usec, L: 0
0113 08:54:36.109844 (+ 2961us) write_op.cc:270] Start()
0113 08:54:36.109877 (+ 33us) write_op.cc:276] Timestamp: P: 1736758473487354 usec, L: 0
0113 08:54:36.111212 (+ 1335us) write_op.cc:270] Start()
0113 08:54:36.111240 (+ 28us) write_op.cc:276] Timestamp: P: 1736758473501103 usec, L: 0
0113 08:54:36.122118 (+ 10878us) write_op.cc:270] Start()
0113 08:54:36.122156 (+ 38us) write_op.cc:276] Timestamp: P: 1736758473536859 usec, L: 0
0113 08:54:36.122431 (+ 275us) write_op.cc:270] Start()
0113 08:54:36.122458 (+ 27us) write_op.cc:276] Timestamp: P: 1736758473597607 usec, L: 0
0113 08:54:36.122695 (+ 237us) write_op.cc:270] Start()
0113 08:54:36.122715 (+ 20us) write_op.cc:276] Timestamp: P: 1736758473620390 usec, L: 0
0113 08:54:36.125952 (+ 3237us) write_op.cc:270] Start()
0113 08:54:36.125990 (+ 38us) write_op.cc:276] Timestamp: P: 1736758473624278 usec, L: 0
0113 08:54:36.126244 (+ 254us) write_op.cc:270] Start()
0113 08:54:36.126265 (+ 21us) write_op.cc:276] Timestamp: P: 1736758473643241 usec, L: 0
0113 08:54:36.126473 (+ 208us) write_op.cc:270] Start()
0113 08:54:36.126498 (+ 25us) write_op.cc:276] Timestamp: P: 1736758473671930 usec, L: 0
0113 08:54:36.126728 (+ 230us) write_op.cc:270] Start()
0113 08:54:36.126753 (+ 25us) write_op.cc:276] Timestamp: P: 1736758473702770 usec, L: 0
0113 08:54:36.128844 (+ 2091us) write_op.cc:270] Start()
0113 08:54:36.128873 (+ 29us) write_op.cc:276] Timestamp: P: 1736758473709571 usec, L: 0
0113 08:54:36.131716 (+ 2843us) write_op.cc:270] Start()
0113 08:54:36.131754 (+ 38us) write_op.cc:276] Timestamp: P: 1736758473720201 usec, L: 0
0113 08:54:36.132884 (+ 1130us) write_op.cc:270] Start()
0113 08:54:36.132914 (+ 30us) write_op.cc:276] Timestamp: P: 1736758473726658 usec, L: 0
0113 08:54:36.143404 (+ 10490us) write_op.cc:270] Start()
0113 08:54:36.143447 (+ 43us) write_op.cc:276] Timestamp: P: 1736758473786778 usec, L: 0
0113 08:54:36.153787 (+ 10340us) write_op.cc:270] Start()
0113 08:54:36.153840 (+ 53us) write_op.cc:276] Timestamp: P: 1736758473791087 usec, L: 0
0113 08:54:36.155124 (+ 1284us) write_op.cc:270] Start()
0113 08:54:36.155161 (+ 37us) write_op.cc:276] Timestamp: P: 1736758473795459 usec, L: 0
0113 08:54:36.168366 (+ 13205us) write_op.cc:270] Start()
0113 08:54:36.168419 (+ 53us) write_op.cc:276] Timestamp: P: 1736758473798269 usec, L: 0
0113 08:54:36.168741 (+ 322us) write_op.cc:270] Start()
0113 08:54:36.168786 (+ 45us) write_op.cc:276] Timestamp: P: 1736758473881189 usec, L: 0
0113 08:54:36.169004 (+ 218us) write_op.cc:270] Start()
0113 08:54:36.169030 (+ 26us) write_op.cc:276] Timestamp: P: 1736758473888883 usec, L: 0
0113 08:54:36.192304 (+ 23274us) write_op.cc:270] Start()
0113 08:54:36.192361 (+ 57us) write_op.cc:276] Timestamp: P: 1736758473889587 usec, L: 0
0113 08:54:36.208624 (+ 16263us) write_op.cc:270] Start()
0113 08:54:36.208662 (+ 38us) write_op.cc:276] Timestamp: P: 1736758473907234 usec, L: 0
0113 08:54:36.211831 (+ 3169us) write_op.cc:270] Start()
0113 08:54:36.211867 (+ 36us) write_op.cc:276] Timestamp: P: 1736758473999741 usec, L: 0
0113 08:54:36.223915 (+ 12048us) write_op.cc:270] Start()
0113 08:54:36.223954 (+ 39us) write_op.cc:276] Timestamp: P: 1736758474001590 usec, L: 0
0113 08:54:36.228412 (+ 4458us) write_op.cc:270] Start()
0113 08:54:36.228462 (+ 50us) write_op.cc:276] Timestamp: P: 1736758474052563 usec, L: 0
0113 08:54:36.228725 (+ 263us) write_op.cc:270] Start()
0113 08:54:36.228749 (+ 24us) write_op.cc:276] Timestamp: P: 1736758474053355 usec, L: 0
0113 08:54:36.241030 (+ 12281us) write_op.cc:270] Start()
0113 08:54:36.241061 (+ 31us) write_op.cc:276] Timestamp: P: 1736758474079324 usec, L: 0
0113 08:54:36.243869 (+ 2808us) write_op.cc:270] Start()
0113 08:54:36.243902 (+ 33us) write_op.cc:276] Timestamp: P: 1736758474103765 usec, L: 0
0113 08:54:36.252028 (+ 8126us) write_op.cc:270] Start()
0113 08:54:36.252061 (+ 33us) write_op.cc:276] Timestamp: P: 1736758474136326 usec, L: 0
0113 08:54:36.255554 (+ 3493us) write_op.cc:270] Start()
0113 08:54:36.255595 (+ 41us) write_op.cc:276] Timestamp: P: 1736758474137099 usec, L: 0
0113 08:54:36.255873 (+ 278us) write_op.cc:270] Start()
0113 08:54:36.255900 (+ 27us) write_op.cc:276] Timestamp: P: 1736758474137784 usec, L: 0
0113 08:54:36.265655 (+ 9755us) write_op.cc:270] Start()
0113 08:54:36.265695 (+ 40us) write_op.cc:276] Timestamp: P: 1736758474176575 usec, L: 0
0113 08:54:36.266176 (+ 481us) write_op.cc:270] Start()
0113 08:54:36.266202 (+ 26us) write_op.cc:276] Timestamp: P: 1736758474220233 usec, L: 0
0113 08:54:36.266446 (+ 244us) write_op.cc:270] Start()
0113 08:54:36.266470 (+ 24us) write_op.cc:276] Timestamp: P: 1736758474221744 usec, L: 0
0113 08:54:36.266713 (+ 243us) write_op.cc:270] Start()
0113 08:54:36.266737 (+ 24us) write_op.cc:276] Timestamp: P: 1736758474240523 usec, L: 0
0113 08:54:36.268520 (+ 1783us) write_op.cc:270] Start()
0113 08:54:36.268551 (+ 31us) write_op.cc:276] Timestamp: P: 1736758474258113 usec, L: 0
0113 08:54:36.268818 (+ 267us) write_op.cc:270] Start()
0113 08:54:36.268844 (+ 26us) write_op.cc:276] Timestamp: P: 1736758474286141 usec, L: 0
0113 08:54:36.299826 (+ 30982us) write_op.cc:270] Start()
0113 08:54:36.299864 (+ 38us) write_op.cc:276] Timestamp: P: 1736758474287896 usec, L: 0
0113 08:54:36.308659 (+ 8795us) write_op.cc:270] Start()
0113 08:54:36.308688 (+ 29us) write_op.cc:276] Timestamp: P: 1736758474307846 usec, L: 0
0113 08:54:36.313906 (+ 5218us) write_op.cc:270] Start()
0113 08:54:36.313946 (+ 40us) write_op.cc:276] Timestamp: P: 1736758474350735 usec, L: 0
0113 08:54:36.324987 (+ 11041us) write_op.cc:270] Start()
0113 08:54:36.325027 (+ 40us) write_op.cc:276] Timestamp: P: 1736758474372215 usec, L: 0
0113 08:54:36.325660 (+ 633us) write_op.cc:270] Start()
0113 08:54:36.325685 (+ 25us) write_op.cc:276] Timestamp: P: 1736758474373593 usec, L: 0
0113 08:54:36.327786 (+ 2101us) write_op.cc:270] Start()
0113 08:54:36.327816 (+ 30us) write_op.cc:276] Timestamp: P: 1736758474378409 usec, L: 0
0113 08:54:36.328029 (+ 213us) write_op.cc:270] Start()
0113 08:54:36.328050 (+ 21us) write_op.cc:276] Timestamp: P: 1736758474442540 usec, L: 0
0113 08:54:36.328223 (+ 173us) write_op.cc:270] Start()
0113 08:54:36.328245 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474476138 usec, L: 0
0113 08:54:36.338593 (+ 10348us) write_op.cc:270] Start()
0113 08:54:36.338635 (+ 42us) write_op.cc:276] Timestamp: P: 1736758474486151 usec, L: 0
0113 08:54:36.338968 (+ 333us) write_op.cc:270] Start()
0113 08:54:36.338998 (+ 30us) write_op.cc:276] Timestamp: P: 1736758474510206 usec, L: 0
0113 08:54:36.339167 (+ 169us) write_op.cc:270] Start()
0113 08:54:36.339190 (+ 23us) write_op.cc:276] Timestamp: P: 1736758474551591 usec, L: 0
0113 08:54:36.343472 (+ 4282us) write_op.cc:270] Start()
0113 08:54:36.343507 (+ 35us) write_op.cc:276] Timestamp: P: 1736758474554115 usec, L: 0
0113 08:54:36.343761 (+ 254us) write_op.cc:270] Start()
0113 08:54:36.343787 (+ 26us) write_op.cc:276] Timestamp: P: 1736758474554875 usec, L: 0
0113 08:54:36.359028 (+ 15241us) write_op.cc:270] Start()
0113 08:54:36.359069 (+ 41us) write_op.cc:276] Timestamp: P: 1736758474597538 usec, L: 0
0113 08:54:36.376459 (+ 17390us) write_op.cc:270] Start()
0113 08:54:36.376508 (+ 49us) write_op.cc:276] Timestamp: P: 1736758474627109 usec, L: 0
0113 08:54:36.379468 (+ 2960us) write_op.cc:270] Start()
0113 08:54:36.379498 (+ 30us) write_op.cc:276] Timestamp: P: 1736758474627804 usec, L: 0
0113 08:54:36.387122 (+ 7624us) write_op.cc:270] Start()
0113 08:54:36.387161 (+ 39us) write_op.cc:276] Timestamp: P: 1736758474639801 usec, L: 0
0113 08:54:36.390343 (+ 3182us) write_op.cc:270] Start()
0113 08:54:36.390391 (+ 48us) write_op.cc:276] Timestamp: P: 1736758474687345 usec, L: 0
0113 08:54:36.390633 (+ 242us) write_op.cc:270] Start()
0113 08:54:36.390655 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474704364 usec, L: 0
0113 08:54:36.390881 (+ 226us) write_op.cc:270] Start()
0113 08:54:36.390903 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474706861 usec, L: 0
0113 08:54:36.391091 (+ 188us) write_op.cc:270] Start()
0113 08:54:36.391113 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474716255 usec, L: 0
0113 08:54:36.391303 (+ 190us) write_op.cc:270] Start()
0113 08:54:36.391325 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474730986 usec, L: 0
0113 08:54:36.391560 (+ 235us) write_op.cc:270] Start()
0113 08:54:36.391581 (+ 21us) write_op.cc:276] Timestamp: P: 1736758474760980 usec, L: 0
0113 08:54:36.391780 (+ 199us) write_op.cc:270] Start()
0113 08:54:36.391800 (+ 20us) write_op.cc:276] Timestamp: P: 1736758474782614 usec, L: 0
0113 08:54:36.391977 (+ 177us) write_op.cc:270] Start()
0113 08:54:36.391996 (+ 19us) write_op.cc:276] Timestamp: P: 1736758474837140 usec, L: 0
0113 08:54:36.392169 (+ 173us) write_op.cc:270] Start()
0113 08:54:36.392192 (+ 23us) write_op.cc:276] Timestamp: P: 1736758474859603 usec, L: 0
0113 08:54:36.392368 (+ 176us) write_op.cc:270] Start()
0113 08:54:36.392395 (+ 27us) write_op.cc:276] Timestamp: P: 1736758474864472 usec, L: 0
0113 08:54:36.392576 (+ 181us) write_op.cc:270] Start()
0113 08:54:36.392598 (+ 22us) write_op.cc:276] Timestamp: P: 173675
I20250113 08:54:37.497978 8754 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:37.509640 8754 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:37.516320 8754 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:37.518558 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 951 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:37.525471 7967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 951 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:37.526664 8583 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:37.740331 8756 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:37.748090 8756 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:37.769419 8661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.3:55187 (request call id 19) took 1556 ms. Trace:
I20250113 08:54:37.786726 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 998 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:37.784592 8756 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:37.789718 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 998 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:37.801599 8581 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:37.808380 8756 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:38.096968 8754 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:38.183897 8754 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:38.187114 8754 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:38.188093 8754 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:38.197804 7967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 998 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:38.209023 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 998 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:38.223670 8581 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:38.228416 8754 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:38.427341 8754 raft_consensus.cc:491] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:38.433653 8754 raft_consensus.cc:513] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:38.434600 8754 leader_election.cc:290] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:38.453526 7967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 1092 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:38.458737 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c2ecb48ca14049a051ee94f7d58cd6" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 1092 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:38.473729 8581 leader_election.cc:304] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:38.568218 8754 raft_consensus.cc:2743] T 44c2ecb48ca14049a051ee94f7d58cd6 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:38.568617 8754 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:38.568796 8754 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:38.569573 8754 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:38.570118 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 998 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:38.579492 7968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 998 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:38.586553 8583 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:38.588232 8754 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:37.781316 8661 rpcz_store.cc:276] 0113 08:54:36.178400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:36.178502 (+ 102us) service_pool.cc:225] Handling call
0113 08:54:36.327333 (+148831us) raft_consensus.cc:1466] Updating replica for 255 ops
0113 08:54:36.489427 (+162094us) raft_consensus.cc:1529] Early marking committed up to index 743
0113 08:54:36.489644 (+ 217us) raft_consensus.cc:1534] Triggering prepare for 255 ops
0113 08:54:36.529156 (+ 39512us) write_op.cc:270] Start()
0113 08:54:36.529206 (+ 50us) write_op.cc:276] Timestamp: P: 1736758470335575 usec, L: 0
0113 08:54:36.531537 (+ 2331us) write_op.cc:270] Start()
0113 08:54:36.532462 (+ 925us) write_op.cc:276] Timestamp: P: 1736758470394956 usec, L: 0
0113 08:54:36.536788 (+ 4326us) write_op.cc:270] Start()
0113 08:54:36.536824 (+ 36us) write_op.cc:276] Timestamp: P: 1736758470396850 usec, L: 0
0113 08:54:36.537054 (+ 230us) write_op.cc:270] Start()
0113 08:54:36.537076 (+ 22us) write_op.cc:276] Timestamp: P: 1736758470408505 usec, L: 0
0113 08:54:36.558366 (+ 21290us) write_op.cc:270] Start()
0113 08:54:36.558403 (+ 37us) write_op.cc:276] Timestamp: P: 1736758470453464 usec, L: 0
0113 08:54:36.560021 (+ 1618us) write_op.cc:270] Start()
0113 08:54:36.560050 (+ 29us) write_op.cc:276] Timestamp: P: 1736758470483601 usec, L: 0
0113 08:54:36.562170 (+ 2120us) write_op.cc:270] Start()
0113 08:54:36.563053 (+ 883us) write_op.cc:276] Timestamp: P: 1736758470523260 usec, L: 0
0113 08:54:36.564477 (+ 1424us) write_op.cc:270] Start()
0113 08:54:36.564509 (+ 32us) write_op.cc:276] Timestamp: P: 1736758470524239 usec, L: 0
0113 08:54:36.567137 (+ 2628us) write_op.cc:270] Start()
0113 08:54:36.567168 (+ 31us) write_op.cc:276] Timestamp: P: 1736758470536848 usec, L: 0
0113 08:54:36.571094 (+ 3926us) write_op.cc:270] Start()
0113 08:54:36.571144 (+ 50us) write_op.cc:276] Timestamp: P: 1736758470609573 usec, L: 0
0113 08:54:36.571925 (+ 781us) write_op.cc:270] Start()
0113 08:54:36.571957 (+ 32us) write_op.cc:276] Timestamp: P: 1736758470622454 usec, L: 0
0113 08:54:36.572129 (+ 172us) write_op.cc:270] Start()
0113 08:54:36.572149 (+ 20us) write_op.cc:276] Timestamp: P: 1736758470628139 usec, L: 0
0113 08:54:36.572286 (+ 137us) write_op.cc:270] Start()
0113 08:54:36.572305 (+ 19us) write_op.cc:276] Timestamp: P: 1736758470631770 usec, L: 0
0113 08:54:36.572486 (+ 181us) write_op.cc:270] Start()
0113 08:54:36.572506 (+ 20us) write_op.cc:276] Timestamp: P: 1736758470686572 usec, L: 0
0113 08:54:36.572643 (+ 137us) write_op.cc:270] Start()
0113 08:54:36.572661 (+ 18us) write_op.cc:276] Timestamp: P: 1736758470687825 usec, L: 0
0113 08:54:36.572804 (+ 143us) write_op.cc:270] Start()
0113 08:54:36.572822 (+ 18us) write_op.cc:276] Timestamp: P: 1736758470703612 usec, L: 0
0113 08:54:36.572959 (+ 137us) write_op.cc:270] Start()
0113 08:54:36.572976 (+ 17us) write_op.cc:276] Timestamp: P: 1736758470751347 usec, L: 0
0113 08:54:36.573136 (+ 160us) write_op.cc:270] Start()
0113 08:54:36.573178 (+ 42us) write_op.cc:276] Timestamp: P: 1736758470779586 usec, L: 0
0113 08:54:36.573306 (+ 128us) write_op.cc:270] Start()
0113 08:54:36.573323 (+ 17us) write_op.cc:276] Timestamp: P: 1736758470824002 usec, L: 0
0113 08:54:36.573458 (+ 135us) write_op.cc:270] Start()
0113 08:54:36.573477 (+ 19us) write_op.cc:276] Timestamp: P: 1736758470844812 usec, L: 0
0113 08:54:36.573615 (+ 138us) write_op.cc:270] Start()
0113 08:54:36.573632 (+ 17us) write_op.cc:276] Timestamp: P: 1736758470848946 usec, L: 0
0113 08:54:36.573755 (+ 123us) write_op.cc:270] Start()
0113 08:54:36.573772 (+ 17us) write_op.cc:276] Timestamp: P: 1736758470902568 usec, L: 0
0113 08:54:36.573904 (+ 132us) write_op.cc:270] Start()
0113 08:54:36.573922 (+ 18us) write_op.cc:276] Timestamp: P: 1736758470909338 usec, L: 0
0113 08:54:36.574061 (+ 139us) write_op.cc:270] Start()
0113 08:54:36.574079 (+ 18us) write_op.cc:276] Timestamp: P: 1736758470909999 usec, L: 0
0113 08:54:36.595119 (+ 21040us) write_op.cc:270] Start()
0113 08:54:36.595164 (+ 45us) write_op.cc:276] Timestamp: P: 1736758470914330 usec, L: 0
0113 08:54:36.614666 (+ 19502us) write_op.cc:270] Start()
0113 08:54:36.614709 (+ 43us) write_op.cc:276] Timestamp: P: 1736758471004731 usec, L: 0
0113 08:54:36.615016 (+ 307us) write_op.cc:270] Start()
0113 08:54:36.615049 (+ 33us) write_op.cc:276] Timestamp: P: 1736758471032261 usec, L: 0
0113 08:54:36.630890 (+ 15841us) write_op.cc:270] Start()
0113 08:54:36.630930 (+ 40us) write_op.cc:276] Timestamp: P: 1736758471035401 usec, L: 0
0113 08:54:36.631456 (+ 526us) write_op.cc:270] Start()
0113 08:54:36.631483 (+ 27us) write_op.cc:276] Timestamp: P: 1736758471036156 usec, L: 0
0113 08:54:36.631662 (+ 179us) write_op.cc:270] Start()
0113 08:54:36.631684 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471072596 usec, L: 0
0113 08:54:36.631831 (+ 147us) write_op.cc:270] Start()
0113 08:54:36.631853 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471117814 usec, L: 0
0113 08:54:36.640445 (+ 8592us) write_op.cc:270] Start()
0113 08:54:36.640483 (+ 38us) write_op.cc:276] Timestamp: P: 1736758471147446 usec, L: 0
0113 08:54:36.645855 (+ 5372us) write_op.cc:270] Start()
0113 08:54:36.645900 (+ 45us) write_op.cc:276] Timestamp: P: 1736758471149816 usec, L: 0
0113 08:54:36.646184 (+ 284us) write_op.cc:270] Start()
0113 08:54:36.646207 (+ 23us) write_op.cc:276] Timestamp: P: 1736758471150601 usec, L: 0
0113 08:54:36.646382 (+ 175us) write_op.cc:270] Start()
0113 08:54:36.646401 (+ 19us) write_op.cc:276] Timestamp: P: 1736758471176673 usec, L: 0
0113 08:54:36.646541 (+ 140us) write_op.cc:270] Start()
0113 08:54:36.646562 (+ 21us) write_op.cc:276] Timestamp: P: 1736758471236573 usec, L: 0
0113 08:54:36.646722 (+ 160us) write_op.cc:270] Start()
0113 08:54:36.646741 (+ 19us) write_op.cc:276] Timestamp: P: 1736758471238535 usec, L: 0
0113 08:54:36.646941 (+ 200us) write_op.cc:270] Start()
0113 08:54:36.646966 (+ 25us) write_op.cc:276] Timestamp: P: 1736758471246135 usec, L: 0
0113 08:54:36.647111 (+ 145us) write_op.cc:270] Start()
0113 08:54:36.647131 (+ 20us) write_op.cc:276] Timestamp: P: 1736758471248938 usec, L: 0
0113 08:54:36.647282 (+ 151us) write_op.cc:270] Start()
0113 08:54:36.647303 (+ 21us) write_op.cc:276] Timestamp: P: 1736758471302273 usec, L: 0
0113 08:54:36.647607 (+ 304us) write_op.cc:270] Start()
0113 08:54:36.647629 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471310297 usec, L: 0
0113 08:54:36.647796 (+ 167us) write_op.cc:270] Start()
0113 08:54:36.647818 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471333378 usec, L: 0
0113 08:54:36.647962 (+ 144us) write_op.cc:270] Start()
0113 08:54:36.647984 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471349201 usec, L: 0
0113 08:54:36.648126 (+ 142us) write_op.cc:270] Start()
0113 08:54:36.648145 (+ 19us) write_op.cc:276] Timestamp: P: 1736758471402895 usec, L: 0
0113 08:54:36.648430 (+ 285us) write_op.cc:270] Start()
0113 08:54:36.648452 (+ 22us) write_op.cc:276] Timestamp: P: 1736758471406496 usec, L: 0
0113 08:54:36.648641 (+ 189us) write_op.cc:270] Start()
0113 08:54:36.648662 (+ 21us) write_op.cc:276] Timestamp: P: 1736758471422927 usec, L: 0
0113 08:54:36.648803 (+ 141us) write_op.cc:270] Start()
0113 08:54:36.648821 (+ 18us) write_op.cc:276] Timestamp: P: 1736758471457310 usec, L: 0
0113 08:54:36.648958 (+ 137us) write_op.cc:270] Start()
0113 08:54:36.648978 (+ 20us) write_op.cc:276] Timestamp: P: 1736758471508670 usec, L: 0
0113 08:54:36.649134 (+ 156us) write_op.cc:270] Start()
0113 08:54:36.649155 (+ 21us) write_op.cc:276] Timestamp: P: 1736758471537674 usec, L: 0
0113 08:54:36.649318 (+ 163us) write_op.cc:270] Start()
0113 08:54:36.649336 (+ 18us) write_op.cc:276] Timestamp: P: 1736758471557708 usec, L: 0
0113 08:54:36.670690 (+ 21354us) write_op.cc:270] Start()
0113 08:54:36.670727 (+ 37us) write_op.cc:276] Timestamp: P: 1736758471566237 usec, L: 0
0113 08:54:36.671286 (+ 559us) write_op.cc:270] Start()
0113 08:54:36.671315 (+ 29us) write_op.cc:276] Timestamp: P: 1736758471580849 usec, L: 0
0113 08:54:36.673101 (+ 1786us) write_op.cc:270] Start()
0113 08:54:36.673129 (+ 28us) write_op.cc:276] Timestamp: P: 1736758471650465 usec, L: 0
0113 08:54:36.677481 (+ 4352us) write_op.cc:270] Start()
0113 08:54:36.677517 (+ 36us) write_op.cc:276] Timestamp: P: 1736758471676336 usec, L: 0
0113 08:54:36.681088 (+ 3571us) write_op.cc:270] Start()
0113 08:54:36.681121 (+ 33us) write_op.cc:276] Timestamp: P: 1736758471695114 usec, L: 0
0113 08:54:36.695145 (+ 14024us) write_op.cc:270] Start()
0113 08:54:36.695185 (+ 40us) write_op.cc:276] Timestamp: P: 1736758471698102 usec, L: 0
0113 08:54:36.697976 (+ 2791us) write_op.cc:270] Start()
0113 08:54:36.698026 (+ 50us) write_op.cc:276] Timestamp: P: 1736758471702073 usec, L: 0
0113 08:54:36.703085 (+ 5059us) write_op.cc:270] Start()
0113 08:54:36.703130 (+ 45us) write_op.cc:276] Timestamp: P: 1736758471745087 usec, L: 0
0113 08:54:36.707451 (+ 4321us) write_op.cc:270] Start()
0113 08:54:36.707490 (+ 39us) write_op.cc:276] Timestamp: P: 1736758471752070 usec, L: 0
0113 08:54:36.714786 (+ 7296us) write_op.cc:270] Start()
0113 08:54:36.717396 (+ 2610us) write_op.cc:276] Timestamp: P: 1736758471752828 usec, L: 0
0113 08:54:36.727239 (+ 9843us) write_op.cc:270] Start()
0113 08:54:36.727279 (+ 40us) write_op.cc:276] Timestamp: P: 1736758471786181 usec, L: 0
0113 08:54:36.730068 (+ 2789us) write_op.cc:270] Start()
0113 08:54:36.730118 (+ 50us) write_op.cc:276] Timestamp: P: 1736758471812441 usec, L: 0
0113 08:54:36.730393 (+ 275us) write_op.cc:270] Start()
0113 08:54:36.730418 (+ 25us) write_op.cc:276] Timestamp: P: 1736758471832471 usec, L: 0
0113 08:54:36.734240 (+ 3822us) write_op.cc:270] Start()
0113 08:54:36.734276 (+ 36us) write_op.cc:276] Timestamp: P: 1736758471852673 usec, L: 0
0113 08:54:36.740931 (+ 6655us) write_op.cc:270] Start()
0113 08:54:36.740973 (+ 42us) write_op.cc:276] Timestamp: P: 1736758471933020 usec, L: 0
0113 08:54:36.747190 (+ 6217us) write_op.cc:270] Start()
0113 08:54:36.747217 (+ 27us) write_op.cc:276] Timestamp: P: 1736758471942848 usec, L: 0
0113 08:54:36.755115 (+ 7898us) write_op.cc:270] Start()
0113 08:54:36.755161 (+ 46us) write_op.cc:276] Timestamp: P: 1736758471961939 usec, L: 0
0113 08:54:36.764722 (+ 9561us) write_op.cc:270] Start()
0113 08:54:36.764757 (+ 35us) write_op.cc:276] Timestamp: P: 1736758471963174 usec, L: 0
0113 08:54:36.768510 (+ 3753us) write_op.cc:270] Start()
0113 08:54:36.768534 (+ 24us) write_op.cc:276] Timestamp: P: 1736758471994522 usec, L: 0
0113 08:54:36.770668 (+ 2134us) write_op.cc:270] Start()
0113 08:54:36.770694 (+ 26us) write_op.cc:276] Timestamp: P: 1736758472036083 usec, L: 0
0113 08:54:36.770899 (+ 205us) write_op.cc:270] Start()
0113 08:54:36.770916 (+ 17us) write_op.cc:276] Timestamp: P: 1736758472044267 usec, L: 0
0113 08:54:36.773154 (+ 2238us) write_op.cc:270] Start()
0113 08:54:36.773181 (+ 27us) write_op.cc:276] Timestamp: P: 1736758472046436 usec, L: 0
0113 08:54:36.775332 (+ 2151us) write_op.cc:270] Start()
0113 08:54:36.775376 (+ 44us) write_op.cc:276] Timestamp: P: 1736758472065439 usec, L: 0
0113 08:54:36.775587 (+ 211us) write_op.cc:270] Start()
0113 08:54:36.775613 (+ 26us) write_op.cc:276] Timestamp: P: 1736758472094364 usec, L: 0
0113 08:54:36.775772 (+ 159us) write_op.cc:270] Start()
0113 08:54:36.775793 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472156585 usec, L: 0
0113 08:54:36.775949 (+ 156us) write_op.cc:270] Start()
0113 08:54:36.775971 (+ 22us) write_op.cc:276] Timestamp: P: 1736758472158052 usec, L: 0
0113 08:54:36.799189 (+ 23218us) write_op.cc:270] Start()
0113 08:54:36.799219 (+ 30us) write_op.cc:276] Timestamp: P: 1736758472167579 usec, L: 0
0113 08:54:36.804361 (+ 5142us) write_op.cc:270] Start()
0113 08:54:36.804398 (+ 37us) write_op.cc:276] Timestamp: P: 1736758472189656 usec, L: 0
0113 08:54:36.804746 (+ 348us) write_op.cc:270] Start()
0113 08:54:36.804771 (+ 25us) write_op.cc:276] Timestamp: P: 1736758472219399 usec, L: 0
0113 08:54:36.804932 (+ 161us) write_op.cc:270] Start()
0113 08:54:36.804952 (+ 20us) write_op.cc:276] Timestamp: P: 1736758472244720 usec, L: 0
0113 08:54:36.805101 (+ 149us) write_op.cc:270] Start()
0113 08:54:36.805123 (+ 22us) write_op.cc:276] Timestamp: P: 1736758472249447 usec, L: 0
0113 08:54:36.805285 (+ 162us) write_op.cc:270] Start()
0113 08:54:36.805305 (+ 20us) write_op.cc:276] Timestamp: P: 1736758472278543 usec, L: 0
0113 08:54:36.805467 (+ 162us) write_op.cc:270] Start()
0113 08:54:36.805486 (+ 19us) write_op.cc:276] Timestamp: P: 1736758472298094 usec, L: 0
0113 08:54:36.806748 (+ 1262us) write_op.cc:270] Start()
0113 08:54:36.806775 (+ 27us) write_op.cc:276] Timestamp: P: 1736758472299794 usec, L: 0
0113 08:54:36.807001 (+ 226us) write_op.cc:270] Start()
0113 08:54:36.807018 (+ 17us) write_op.cc:276] Timestamp: P: 1736758472328952 usec, L: 0
0113 08:54:36.807121 (+ 103us) write_op.cc:270] Start()
0113 08:54:36.807134 (+ 13us) write_op.cc:276] Timestamp: P: 1736758472360059 usec, L: 0
0113 08:54:36.807239 (+ 105us) write_op.cc:270] Start()
0113 08:54:36.807252 (+ 13us) write_op.cc:276] Timestamp: P: 1736758472369322 usec, L: 0
0113 08:54:36.807402 (+ 150us) write_op.cc:270] Start()
0113 08:54:36.807423 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472382264 usec, L: 0
0113 08:54:36.807561 (+ 138us) write_op.cc:270] Start()
0113 08:54:36.807579 (+ 18us) write_op.cc:276] Timestamp: P: 1736758472427761 usec, L: 0
0113 08:54:36.807732 (+ 153us) write_op.cc:270] Start()
0113 08:54:36.807751 (+ 19us) write_op.cc:276] Timestamp: P: 1736758472434277 usec, L: 0
0113 08:54:36.807888 (+ 137us) write_op.cc:270] Start()
0113 08:54:36.807909 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472435099 usec, L: 0
0113 08:54:36.808058 (+ 149us) write_op.cc:270] Start()
0113 08:54:36.808078 (+ 20us) write_op.cc:276] Timestamp: P: 1736758472529539 usec, L: 0
0113 08:54:36.844501 (+ 36423us) write_op.cc:270] Start()
0113 08:54:36.844540 (+ 39us) write_op.cc:276] Timestamp: P: 1736758472538657 usec, L: 0
0113 08:54:36.846415 (+ 1875us) write_op.cc:270] Start()
0113 08:54:36.846450 (+ 35us) write_op.cc:276] Timestamp: P: 1736758472540043 usec, L: 0
0113 08:54:36.846702 (+ 252us) write_op.cc:270] Start()
0113 08:54:36.846726 (+ 24us) write_op.cc:276] Timestamp: P: 1736758472549405 usec, L: 0
0113 08:54:36.846981 (+ 255us) write_op.cc:270] Start()
0113 08:54:36.847004 (+ 23us) write_op.cc:276] Timestamp: P: 1736758472626190 usec, L: 0
0113 08:54:36.847143 (+ 139us) write_op.cc:270] Start()
0113 08:54:36.847163 (+ 20us) write_op.cc:276] Timestamp: P: 1736758472629250 usec, L: 0
0113 08:54:36.847353 (+ 190us) write_op.cc:270] Start()
0113 08:54:36.847380 (+ 27us) write_op.cc:276] Timestamp: P: 1736758472630361 usec, L: 0
0113 08:54:36.847611 (+ 231us) write_op.cc:270] Start()
0113 08:54:36.847632 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472631311 usec, L: 0
0113 08:54:36.847779 (+ 147us) write_op.cc:270] Start()
0113 08:54:36.847795 (+ 16us) write_op.cc:276] Timestamp: P: 1736758472707904 usec, L: 0
0113 08:54:36.847921 (+ 126us) write_op.cc:270] Start()
0113 08:54:36.847940 (+ 19us) write_op.cc:276] Timestamp: P: 1736758472719533 usec, L: 0
0113 08:54:36.857943 (+ 10003us) write_op.cc:270] Start()
0113 08:54:36.857997 (+ 54us) write_op.cc:276] Timestamp: P: 1736758472723940 usec, L: 0
0113 08:54:36.858274 (+ 277us) write_op.cc:270] Start()
0113 08:54:36.858298 (+ 24us) write_op.cc:276] Timestamp: P: 1736758472735313 usec, L: 0
0113 08:54:36.860824 (+ 2526us) write_op.cc:270] Start()
0113 08:54:36.860870 (+ 46us) write_op.cc:276] Timestamp: P: 1736758472806919 usec, L: 0
0113 08:54:36.880941 (+ 20071us) write_op.cc:270] Start()
0113 08:54:36.880971 (+ 30us) write_op.cc:276] Timestamp: P: 1736758472815230 usec, L: 0
0113 08:54:36.881162 (+ 191us) write_op.cc:270] Start()
0113 08:54:36.881183 (+ 21us) write_op.cc:276] Timestamp: P: 1736758472823983 usec, L: 0
0113 08:54:36.901322 (+ 20139us) write_op.cc:270] Start()
0113 08:54:36.901354 (+ 32us) write_op.cc:276] Timestamp: P: 1736758472837617 usec, L: 0
0113 08:54:36.902264 (+ 910us) write_op.cc:270] Start()
0113 08:54:36.902283 (+ 19us) write_op.cc:276] Timestamp: P: 1736758472933056 usec, L: 0
0113 08:54:36.914931 (+ 12648us) write_op.cc:270] Start()
0113 08:54:36.914980 (+ 49us) write_op.cc:276] Timestamp: P: 1736758472939121 usec, L: 0
0113 08:54:36.922174 (+ 7194us) write_op.cc:270] Start()
0113 08:54:36.922211 (+ 37us) write_op.cc:276] Timestamp: P: 1736758472950472 usec, L: 0
0113 08:54:36.922429 (+ 218us) write_op.cc:270] Start()
0113 08:54:36.922446 (+ 17us) write_op.cc:276] Timestamp: P: 1736758472951838 usec, L: 0
0113 08:54:36.926346 (+ 3900us) write_op.cc:270] Start()
0113 08:54:36.926377 (+ 31us) write_op.cc:276] Timestamp: P: 1736758472993297 usec, L: 0
0113 08:54:36.933414 (+ 7037us) write_op.cc:270] Start()
0113 08:54:36.933451 (+ 37us) write_op.cc:276] Timestamp: P: 1736758473035359 usec, L: 0
0113 08:54:36.947100 (+ 13649us) write_op.cc:270] Start()
0113 08:54:36.947138 (+ 38us) write_op.cc:276] Timestamp: P: 1736758473041447 usec, L: 0
0113 08:54:36.953547 (+ 6409us) write_op.cc:270] Start()
0113 08:54:36.953584 (+ 37us) write_op.cc:276] Timestamp: P: 1736758473068232 usec, L: 0
0113 08:54:36.953818 (+ 234us) write_op.cc:270] Start()
0113 08:54:36.953839 (+ 21us) write_op.cc:276] Timestamp: P: 1736758473074459 usec, L: 0
0113 08:54:36.956392 (+ 2553us) write_op.cc:270] Start()
0113 08:54:36.956429 (+ 37us) write_op.cc:276] Timestamp: P: 1736758473088930 usec, L: 0
0113 08:54:36.959223 (+ 2794us) write_op.cc:270] Start()
0113 08:54:36.959251 (+ 28us) write_op.cc:276] Timestamp: P: 1736758473126958 usec, L: 0
0113 08:54:36.973315 (+ 14064us) write_op.cc:270] Start()
0113 08:54:36.973352 (+ 37us) write_op.cc:276] Timestamp: P: 1736758473128957 usec, L: 0
0113 08:54:36.998566 (+ 25214us) write_op.cc:270] Start()
0113 08:54:36.998603 (+ 37us) write_op.cc:276] Timestamp: P: 1736758473170691 usec, L: 0
0113 08:54:36.999800 (+ 1197us) write_op.cc:270] Start()
0113 08:54:36.999831 (+ 31us) write_op.cc:276] Timestamp: P: 1736758473173247 usec, L: 0
0113 08:54:37.000009 (+ 178us) write_op.cc:270] Start()
0113 08:54:37.000027 (+ 18us) write_op.cc:276] Timestamp: P: 1736758473228076 usec, L: 0
0113 08:54:37.008415 (+ 8388us) write_op.cc:270] Start()
0113 08:54:37.008464 (+ 49us) write_op.cc:276] Timestamp: P: 1736758473232120 usec, L: 0
0113 08:54:37.021553 (+ 13089us) write_op.cc:270] Start()
0113 08:54:37.021592 (+ 39us) write_op.cc:276] Timestamp: P: 1736758473232791 usec, L: 0
0113 08:54:37.026695 (+ 5103us) write_op.cc:270] Start()
0113 08:54:37.026727 (+ 32us) write_op.cc:276] Timestamp: P: 1736758473248725 usec, L: 0
0113 08:54:37.026960 (+ 233us) write_op.cc:270] Start()
0113 08:54:37.026989 (+ 29us) write_op.cc:276] Timestamp: P: 1736758473280441 usec, L: 0
0113 08:54:37.027145 (+ 156us) write_op.cc:270] Start()
0113 08:54:37.027165 (+ 20us) write_op.cc:276] Timestamp: P: 1736758473321181 usec, L: 0
0113 08:54:37.059123 (+ 31958us) write_op.cc:270] Start()
0113 08:54:37.059154 (+ 31us) write_op.cc:276] Timestamp: P: 1736758473322668 usec, L: 0
0113 08:54:37.069818 (+ 10664us) write_op.cc:270] Start()
0113 08:54:37.069852 (+ 34us) write_op.cc:276] Timestamp: P: 1736758473345059 usec, L: 0
0113 08:54:37.070077 (+ 225us) write_op.cc:270] Start()
0113 08:54:37.070104 (+ 27us) write_op.cc:276] Timestamp: P: 1736758473395290 usec, L: 0
0113 08:54:37.070290 (+ 186us) write_op.cc:270] Start()
0113 08:54:37.070341 (+ 51us) write_op.cc:276] Timestamp: P: 1736758473399267 usec, L: 0
0113 08:54:37.070498 (+ 157us) write_op.cc:270] Start()
0113 08:54:37.070523 (+ 25us) write_op.cc:276] Timestamp: P: 1736758473415318 usec, L: 0
0113 08:54:37.070723 (+ 200us) write_op.cc:270] Start()
0113 08:54:37.070747 (+ 24us) write_op.cc:276] Timestamp: P: 1736758473450598 usec, L: 0
0113 08:54:37.070934 (+ 187us) write_op.cc:270] Start()
0113 08:54:37.070956 (+ 22us) write_op.cc:276] Timestamp: P: 1736758473474951 usec, L: 0
0113 08:54:37.071091 (+ 135us) write_op.cc:270] Start()
0113 08:54:37.071110 (+ 19us) write_op.cc:276] Timestamp: P: 1736758473475953 usec, L: 0
0113 08:54:37.071256 (+ 146us) write_op.cc:270] Start()
0113 08:54:37.071275 (+ 19us) write_op.cc:276] Timestamp: P: 1736758473492308 usec, L: 0
0113 08:54:37.071429 (+ 154us) write_op.cc:270] Start()
0113 08:54:37.071449 (+ 20us) write_op.cc:276] Timestamp: P: 1736758473551717 usec, L: 0
0113 08:54:37.071600 (+ 151us) write_op.cc:270] Start()
0113 08:54:37.071621 (+ 21us) write_op.cc:276] Timestamp: P: 1736758473611289 usec, L: 0
0113 08:54:37.071757 (+ 136us) write_op.cc:270] Start()
0113 08:54:37.071775 (+ 18us) write_op.cc:276] Timestamp: P: 1736758473617966 usec, L: 0
0113 08:54:37.071915 (+ 140us) write_op.cc:270] Start()
0113 08:54:37.071948 (+ 33us) write_op.cc:276] Timestamp: P: 1736758473618797 usec, L: 0
0113 08:54:37.072112 (+ 164us) write_op.cc:270] Start()
0113 08:54:37.072133 (+ 21us) write_op.cc:276] Timestamp: P: 1736758473621946 usec, L: 0
0113 08:54:37.072289 (+ 156us) write_op.cc:270] Start()
0113 08:54:37.072308 (+ 19us) write_op.cc:276] Timestamp: P: 1736758473667389 usec, L: 0
0113 08:54:37.072449 (+ 141us) write_op.cc:270] Start()
0113 08:54:37.072466 (+ 17us) write_op.cc:276] Timestamp: P: 1736758473701493 usec, L: 0
0113 08:54:37.072635 (+ 169us) write_op.cc:270] Start()
0113 08:54:37.072658 (+ 23us) write_op.cc:276] Timestamp: P: 1736758473710133 usec, L: 0
0113 08:54:37.078037 (+ 5379us) write_op.cc:270] Start()
0113 08:54:37.078065 (+ 28us) write_op.cc:276] Timestamp: P: 1736758473716522 usec, L: 0
0113 08:54:37.105289 (+ 27224us) write_op.cc:270] Start()
0113 08:54:37.105325 (+ 36us) write_op.cc:276] Timestamp: P: 1736758473720782 usec, L: 0
0113 08:54:37.115884 (+ 10559us) write_op.cc:270] Start()
0113 08:54:37.115904 (+ 20us) write_op.cc:276] Timestamp: P: 1736758473788218 usec, L: 0
0113 08:54:37.117109 (+ 1205us) write_op.cc:270] Start()
0113 08:54:37.117137 (+ 28us) write_op.cc:276] Timestamp: P: 1736758473789351 usec, L: 0
0113 08:54:37.123387 (+ 6250us) write_op.cc:270] Start()
0113 08:54:37.123422 (+ 35us) write_op.cc:276] Timestamp: P: 1736758473792365 usec, L: 0
0113 08:54:37.130384 (+ 6962us) write_op.cc:270] Start()
0113 08:54:37.130414 (+ 30us) write_op.cc:276] Timestamp: P: 1736758473796089 usec, L: 0
0113 08:54:37.130606 (+ 192us) write_op.cc:270] Start()
0113 08:54:37.130622 (+ 16us) write_op.cc:276] Timestamp: P: 1736758473881308 usec, L: 0
0113 08:54:37.167974 (+ 37352us) write_op.cc:270] Start()
0113 08:54:37.168009 (+ 35us) write_op.cc:276] Timestamp: P: 1736758473883042 usec, L: 0
0113 08:54:37.182859 (+ 14850us) write_op.cc:270] Start()
0113 08:54:37.182901 (+ 42us) write_op.cc:276] Timestamp: P: 1736758473886087 usec, L: 0
0113 08:54:37.183154 (+ 253us) write_op.cc:270] Start()
0113 08:54:37.183177 (+ 23us) write_op.cc:276] Timestamp: P: 1736758473904889 usec, L: 0
0113 08:54:37.183356 (+ 179us) write_op.cc:270] Start()
0113 08:54:37.183382 (+ 26us) write_op.cc:276] Timestamp: P: 1736758473998374 usec, L: 0
0113 08:54:37.183527 (+ 145us) write_op.cc:270] Start()
0113 08:54:37.183548 (+ 21us) write_op.cc:276] Timestamp: P: 1736758474021000 usec, L: 0
0113 08:54:37.183720 (+ 172us) write_op.cc:270] Start()
0113 08:54:37.183742 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474021866 usec, L: 0
0113 08:54:37.183904 (+ 162us) write_op.cc:270] Start()
0113 08:54:37.183926 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474022603 usec, L: 0
0113 08:54:37.184082 (+ 156us) write_op.cc:270] Start()
0113 08:54:37.184100 (+ 18us) write_op.cc:276] Timestamp: P: 1736758474078707 usec, L: 0
0113 08:54:37.184241 (+ 141us) write_op.cc:270] Start()
0113 08:54:37.184260 (+ 19us) write_op.cc:276] Timestamp: P: 1736758474105361 usec, L: 0
0113 08:54:37.184425 (+ 165us) write_op.cc:270] Start()
0113 08:54:37.184448 (+ 23us) write_op.cc:276] Timestamp: P: 1736758474116424 usec, L: 0
0113 08:54:37.184607 (+ 159us) write_op.cc:270] Start()
0113 08:54:37.184624 (+ 17us) write_op.cc:276] Timestamp: P: 1736758474117167 usec, L: 0
0113 08:54:37.184728 (+ 104us) write_op.cc:270] Start()
0113 08:54:37.184741 (+ 13us) write_op.cc:276] Timestamp: P: 1736758474161390 usec, L: 0
0113 08:54:37.184830 (+ 89us) write_op.cc:270] Start()
0113 08:54:37.184841 (+ 11us) write_op.cc:276] Timestamp: P: 1736758474165774 usec, L: 0
0113 08:54:37.184973 (+ 132us) write_op.cc:270] Start()
0113 08:54:37.184992 (+ 19us) write_op.cc:276] Timestamp: P: 1736758474221075 usec, L: 0
0113 08:54:37.185140 (+ 148us) write_op.cc:270] Start()
0113 08:54:37.185158 (+ 18us) write_op.cc:276] Timestamp: P: 1736758474222076 usec, L: 0
0113 08:54:37.185343 (+ 185us) write_op.cc:270] Start()
0113 08:54:37.185365 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474239820 usec, L: 0
0113 08:54:37.185518 (+ 153us) write_op.cc:270] Start()
0113 08:54:37.185536 (+ 18us) write_op.cc:276] Timestamp: P: 1736758474257459 usec, L: 0
0113 08:54:37.185741 (+ 205us) write_op.cc:270] Start()
0113 08:54:37.185763 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474285437 usec, L: 0
0113 08:54:37.185914 (+ 151us) write_op.cc:270] Start()
0113 08:54:37.185936 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474290162 usec, L: 0
0113 08:54:37.186132 (+ 196us) write_op.cc:270] Start()
0113 08:54:37.186154 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474307832 usec, L: 0
0113 08:54:37.186318 (+ 164us) write_op.cc:270] Start()
0113 08:54:37.186334 (+ 16us) write_op.cc:276] Timestamp: P: 1736758474371363 usec, L: 0
0113 08:54:37.186468 (+ 134us) write_op.cc:270] Start()
0113 08:54:37.186481 (+ 13us) write_op.cc:276] Timestamp: P: 1736758474372133 usec, L: 0
0113 08:54:37.186602 (+ 121us) write_op.cc:270] Start()
0113 08:54:37.186623 (+ 21us) write_op.cc:276] Timestamp: P: 1736758474373011 usec, L: 0
0113 08:54:37.186770 (+ 147us) write_op.cc:270] Start()
0113 08:54:37.186789 (+ 19us) write_op.cc:276] Timestamp: P: 1736758474377867 usec, L: 0
0113 08:54:37.197744 (+ 10955us) write_op.cc:270] Start()
0113 08:54:37.197792 (+ 48us) write_op.cc:276] Timestamp: P: 1736758474445643 usec, L: 0
0113 08:54:37.212571 (+ 14779us) write_op.cc:270] Start()
0113 08:54:37.212614 (+ 43us) write_op.cc:276] Timestamp: P: 1736758474477875 usec, L: 0
0113 08:54:37.231715 (+ 19101us) write_op.cc:270] Start()
0113 08:54:37.231747 (+ 32us) write_op.cc:276] Timestamp: P: 1736758474481511 usec, L: 0
0113 08:54:37.232002 (+ 255us) write_op.cc:270] Start()
0113 08:54:37.232025 (+ 23us) write_op.cc:276] Timestamp: P: 1736758474511079 usec, L: 0
0113 08:54:37.232178 (+ 153us) write_op.cc:270] Start()
0113 08:54:37.232197 (+ 19us) write_op.cc:276] Timestamp: P: 1736758474549812 usec, L: 0
0113 08:54:37.232353 (+ 156us) write_op.cc:270] Start()
0113 08:54:37.232373 (+ 20us) write_op.cc:276] Timestamp: P: 1736758474550520 usec, L: 0
0113 08:54:37.232538 (+ 165us) write_op.cc:270] Start()
0113 08:54:37.232596 (+ 58us) write_op.cc:276] Timestamp: P: 1736758474554895 usec, L: 0
0113 08:54:37.232740 (+ 144us) write_op.cc:270] Start()
0113 08:54:37.232759 (+ 19us) write_op.cc:276] Timestamp: P: 1736758474597606 usec, L: 0
0113 08:54:37.232939 (+ 180us) write_op.cc:270] Start()
0113 08:54:37.232960 (+ 21us) write_op.cc:276] Timestamp: P: 1736758474626764 usec, L: 0
0113 08:54:37.233143 (+ 183us) write_op.cc:270] Start()
0113 08:54:37.233170 (+ 27us) write_op.cc:276] Timestamp: P: 1736758474628097 usec, L: 0
0113 08:54:37.233314 (+ 144us) write_op.cc:270] Start()
0113 08:54:37.233334 (+ 20us) write_op.cc:276] Timestamp: P: 1736758474634420 usec, L: 0
0113 08:54:37.233517 (+ 183us) write_op.cc:270] Start()
0113 08:54:37.233538 (+ 21us) write_op.cc:276] Timestamp: P: 1736758474687999 usec, L: 0
0113 08:54:37.233716 (+ 178us) write_op.cc:270] Start()
0113 08:54:37.233737 (+ 21us) write_op.cc:276] Timestamp: P: 1736758474701062 usec, L: 0
0113 08:54:37.233898 (+ 161us) write_op.cc:270] Start()
0113 08:54:37.233919 (+ 21us) write_op.cc:276] Timestamp: P: 1736758474704495 usec, L: 0
0113 08:54:37.234212 (+ 293us) write_op.cc:270] Start()
0113 08:54:37.234235 (+ 23us) write_op.cc:276] Timestamp: P: 1736758474716363 usec, L: 0
0113 08:54:37.234438 (+ 203us) write_op.cc:270] Start()
0113 08:54:37.234459 (+ 21us) write_op.cc:276] Timestamp: P: 1736758474732043 usec, L: 0
0113 08:54:37.234656 (+ 197us) write_op.cc:270] Start()
0113 08:54:37.234679 (+ 23us) write_op.cc:276] Timestamp: P: 1736758474771296 usec, L: 0
0113 08:54:37.234874 (+ 195us) write_op.cc:270] Start()
0113 08:54:37.234896 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474783377 usec, L: 0
0113 08:54:37.235098 (+ 202us) write_op.cc:270] Start()
0113 08:54:37.235126 (+ 28us) write_op.cc:276] Timestamp: P: 1736758474837150 usec, L: 0
0113 08:54:37.235292 (+ 166us) write_op.cc:270] Start()
0113 08:54:37.235314 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474858033 usec, L: 0
0113 08:54:37.235512 (+ 198us) write_op.cc:270] Start()
0113 08:54:37.235534 (+ 22us) write_op.cc:276] Timestamp: P: 1736758474862947 usec, L: 0
0113 08:54:37.235708 (+ 174us) write_op.cc:270] Start()
0113 08:54:37.235730 (+ 22us) write_op.cc:276] Timestamp: P: 173675
W20250113 08:54:39.118640 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 126: this message will repeat every 5th retry.
I20250113 08:54:39.736670 8765 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:39.736943 8765 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:39.817358 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1123 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:39.818158 8765 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:39.839640 7968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1123 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:39.856503 8583 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:39.856874 8765 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:39.897493 8583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.123.2:45341, user_credentials={real_user=slave}} blocked reactor thread for 52373.7us
W20250113 08:54:39.996958 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
I20250113 08:54:40.301568 8765 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:40.316020 8765 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:40.322239 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1171 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:40.322712 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1171 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:40.323843 8765 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:40.348071 8581 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:40.662865 8765 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:42.387490 8662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.3:55187 (request call id 17) took 6739 ms. Trace:
I20250113 08:54:42.392475 8757 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:42.420604 8615 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37056 (request call id 40) took 4053 ms. Trace:
I20250113 08:54:42.401858 8757 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:42.427618 8615 rpcz_store.cc:276] 0113 08:54:38.359731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:38.360193 (+ 462us) service_pool.cc:225] Handling call
0113 08:54:42.395550 (+4035357us) spinlock_profiling.cc:243] Waited 3.99 s on lock 0x61600001a554. stack: 00007f4647092fd6 00007f4647092d90 00007f464607248d 000000000096d544 000000000096d50d 000000000096ceda 00007f4652905fd0 00007f4656642b3f 00007f4650f9d8e6 00007f4650f9d615 00007f464ddfa30e 00007f464ddf91ee 00007f464ddff425 00007f464ddffd4f 00007f464ddffb78 0000000000a71ecc
0113 08:54:42.413574 (+ 18024us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9182500096}
I20250113 08:54:42.473089 8757 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:42.475565 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 970 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:42.479388 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 970 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:42.487300 8583 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:42.488392 8757 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:42.628300 8586 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap replaying log segment 1/1 (74.2K/110.5K this segment, stats: ops{read=136 overwritten=0 applied=136 ignored=0} inserts{seen=1079 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250113 08:54:42.814540 8757 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:42.814780 8757 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:42.817759 7968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 970 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:42.819806 8757 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:42.829794 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 970 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:42.835366 8581 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:42.847172 8757 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:43.200965 8757 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:43.207405 8757 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:43.209242 8757 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:43.209172 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 970 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:43.214789 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 970 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:43.216198 8581 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:43.216583 8757 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:43.449172 8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.2:42655 (request call id 42) took 1003 ms. Trace:
I20250113 08:54:43.466231 8656 rpcz_store.cc:276] 0113 08:54:42.422451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:42.422522 (+ 71us) service_pool.cc:225] Handling call
0113 08:54:43.425966 (+1003444us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:54:43.479069 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 131: this message will repeat every 5th retry.
I20250113 08:54:43.912650 8586 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=201 overwritten=0 applied=200 ignored=0} inserts{seen=1596 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 08:54:43.913398 8586 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap complete.
I20250113 08:54:43.915338 8586 ts_tablet_manager.cc:1397] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 7.780s user 0.431s sys 0.061s
I20250113 08:54:43.923990 8586 raft_consensus.cc:357] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:43.924778 8586 raft_consensus.cc:738] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:43.925207 8586 consensus_queue.cc:260] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 200, Last appended: 1.201, Last appended by leader: 201, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:44.011240 8586 ts_tablet_manager.cc:1428] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.096s user 0.014s sys 0.001s
I20250113 08:54:44.020058 8586 tablet_bootstrap.cc:492] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap starting.
I20250113 08:54:44.216403 8774 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:44.216635 8774 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:44.220439 8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 970 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:44.219583 8774 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:44.234000 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 970 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:44.246392 8583 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:44.247257 8774 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:44.300609 8774 raft_consensus.cc:491] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:44.300837 8774 raft_consensus.cc:513] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:44.302893 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35e4ecbe2cc424089eaa5e0249df483" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 201 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:44.305421 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35e4ecbe2cc424089eaa5e0249df483" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 201 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:44.319068 8774 leader_election.cc:290] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:44.347820 8581 leader_election.cc:304] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:44.364979 8774 raft_consensus.cc:2743] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:42.392407 8662 rpcz_store.cc:276] 0113 08:54:35.648160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:35.648226 (+ 66us) service_pool.cc:225] Handling call
0113 08:54:36.152253 (+504027us) raft_consensus.cc:1466] Updating replica for 768 ops
0113 08:54:36.863099 (+710846us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:54:36.863102 (+ 3us) write_op.cc:454] Released schema lock
0113 08:54:36.863451 (+ 349us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:54:36.863454 (+ 3us) write_op.cc:454] Released schema lock
0113 08:54:36.864465 (+ 1011us) raft_consensus.cc:1529] Early marking committed up to index 0
0113 08:54:36.864471 (+ 6us) raft_consensus.cc:1534] Triggering prepare for 768 ops
0113 08:54:36.902010 (+ 37539us) write_op.cc:270] Start()
0113 08:54:36.902036 (+ 26us) write_op.cc:276] Timestamp: P: 1736758458749368 usec, L: 0
0113 08:54:37.041854 (+139818us) write_op.cc:270] Start()
0113 08:54:37.041899 (+ 45us) write_op.cc:276] Timestamp: P: 1736758458753266 usec, L: 0
0113 08:54:37.046096 (+ 4197us) write_op.cc:270] Start()
0113 08:54:37.046133 (+ 37us) write_op.cc:276] Timestamp: P: 1736758458806280 usec, L: 0
0113 08:54:37.068982 (+ 22849us) write_op.cc:270] Start()
0113 08:54:37.069019 (+ 37us) write_op.cc:276] Timestamp: P: 1736758458809841 usec, L: 0
0113 08:54:37.077448 (+ 8429us) write_op.cc:270] Start()
0113 08:54:37.077494 (+ 46us) write_op.cc:276] Timestamp: P: 1736758458836257 usec, L: 0
0113 08:54:37.105322 (+ 27828us) write_op.cc:270] Start()
0113 08:54:37.105352 (+ 30us) write_op.cc:276] Timestamp: P: 1736758458854670 usec, L: 0
0113 08:54:37.106860 (+ 1508us) write_op.cc:270] Start()
0113 08:54:37.106902 (+ 42us) write_op.cc:276] Timestamp: P: 1736758458863155 usec, L: 0
0113 08:54:37.115101 (+ 8199us) write_op.cc:270] Start()
0113 08:54:37.115146 (+ 45us) write_op.cc:276] Timestamp: P: 1736758458886767 usec, L: 0
0113 08:54:37.115449 (+ 303us) write_op.cc:270] Start()
0113 08:54:37.115475 (+ 26us) write_op.cc:276] Timestamp: P: 1736758458907242 usec, L: 0
0113 08:54:37.117233 (+ 1758us) write_op.cc:270] Start()
0113 08:54:37.117259 (+ 26us) write_op.cc:276] Timestamp: P: 1736758458943828 usec, L: 0
0113 08:54:37.126915 (+ 9656us) write_op.cc:270] Start()
0113 08:54:37.126955 (+ 40us) write_op.cc:276] Timestamp: P: 1736758458946167 usec, L: 0
0113 08:54:37.131260 (+ 4305us) write_op.cc:270] Start()
0113 08:54:37.131287 (+ 27us) write_op.cc:276] Timestamp: P: 1736758458976105 usec, L: 0
0113 08:54:37.133561 (+ 2274us) write_op.cc:270] Start()
0113 08:54:37.133598 (+ 37us) write_op.cc:276] Timestamp: P: 1736758458978043 usec, L: 0
0113 08:54:37.137456 (+ 3858us) write_op.cc:270] Start()
0113 08:54:37.137491 (+ 35us) write_op.cc:276] Timestamp: P: 1736758458994848 usec, L: 0
0113 08:54:37.139144 (+ 1653us) write_op.cc:270] Start()
0113 08:54:37.139161 (+ 17us) write_op.cc:276] Timestamp: P: 1736758458998125 usec, L: 0
0113 08:54:37.139324 (+ 163us) write_op.cc:270] Start()
0113 08:54:37.139341 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459044169 usec, L: 0
0113 08:54:37.140162 (+ 821us) write_op.cc:270] Start()
0113 08:54:37.140224 (+ 62us) write_op.cc:276] Timestamp: P: 1736758459048294 usec, L: 0
0113 08:54:37.166118 (+ 25894us) write_op.cc:270] Start()
0113 08:54:37.166171 (+ 53us) write_op.cc:276] Timestamp: P: 1736758459050433 usec, L: 0
0113 08:54:37.166469 (+ 298us) write_op.cc:270] Start()
0113 08:54:37.166498 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459053056 usec, L: 0
0113 08:54:37.169133 (+ 2635us) write_op.cc:270] Start()
0113 08:54:37.169170 (+ 37us) write_op.cc:276] Timestamp: P: 1736758459089856 usec, L: 0
0113 08:54:37.191825 (+ 22655us) write_op.cc:270] Start()
0113 08:54:37.191876 (+ 51us) write_op.cc:276] Timestamp: P: 1736758459104504 usec, L: 0
0113 08:54:37.192100 (+ 224us) write_op.cc:270] Start()
0113 08:54:37.193977 (+ 1877us) write_op.cc:276] Timestamp: P: 1736758459111303 usec, L: 0
0113 08:54:37.215443 (+ 21466us) write_op.cc:270] Start()
0113 08:54:37.215484 (+ 41us) write_op.cc:276] Timestamp: P: 1736758459118279 usec, L: 0
0113 08:54:37.239853 (+ 24369us) write_op.cc:270] Start()
0113 08:54:37.239882 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459134049 usec, L: 0
0113 08:54:37.240053 (+ 171us) write_op.cc:270] Start()
0113 08:54:37.240075 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459169165 usec, L: 0
0113 08:54:37.255506 (+ 15431us) write_op.cc:270] Start()
0113 08:54:37.255534 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459184450 usec, L: 0
0113 08:54:37.255720 (+ 186us) write_op.cc:270] Start()
0113 08:54:37.255739 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459188768 usec, L: 0
0113 08:54:37.255896 (+ 157us) write_op.cc:270] Start()
0113 08:54:37.255916 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459195148 usec, L: 0
0113 08:54:37.256100 (+ 184us) write_op.cc:270] Start()
0113 08:54:37.256121 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459230032 usec, L: 0
0113 08:54:37.257401 (+ 1280us) write_op.cc:270] Start()
0113 08:54:37.257435 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459233556 usec, L: 0
0113 08:54:37.261447 (+ 4012us) write_op.cc:270] Start()
0113 08:54:37.261493 (+ 46us) write_op.cc:276] Timestamp: P: 1736758459260032 usec, L: 0
0113 08:54:37.277684 (+ 16191us) write_op.cc:270] Start()
0113 08:54:37.277718 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459261562 usec, L: 0
0113 08:54:37.277947 (+ 229us) write_op.cc:270] Start()
0113 08:54:37.290740 (+ 12793us) write_op.cc:276] Timestamp: P: 1736758459290248 usec, L: 0
0113 08:54:37.293410 (+ 2670us) write_op.cc:270] Start()
0113 08:54:37.293447 (+ 37us) write_op.cc:276] Timestamp: P: 1736758459293446 usec, L: 0
0113 08:54:37.295684 (+ 2237us) write_op.cc:270] Start()
0113 08:54:37.295714 (+ 30us) write_op.cc:276] Timestamp: P: 1736758459316401 usec, L: 0
0113 08:54:37.307206 (+ 11492us) write_op.cc:270] Start()
0113 08:54:37.307239 (+ 33us) write_op.cc:276] Timestamp: P: 1736758459317105 usec, L: 0
0113 08:54:37.310100 (+ 2861us) write_op.cc:270] Start()
0113 08:54:37.310148 (+ 48us) write_op.cc:276] Timestamp: P: 1736758459345580 usec, L: 0
0113 08:54:37.332554 (+ 22406us) write_op.cc:270] Start()
0113 08:54:37.332599 (+ 45us) write_op.cc:276] Timestamp: P: 1736758459360699 usec, L: 0
0113 08:54:37.335609 (+ 3010us) write_op.cc:270] Start()
0113 08:54:37.335637 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459365945 usec, L: 0
0113 08:54:37.342923 (+ 7286us) write_op.cc:270] Start()
0113 08:54:37.342966 (+ 43us) write_op.cc:276] Timestamp: P: 1736758459387737 usec, L: 0
0113 08:54:37.343475 (+ 509us) write_op.cc:270] Start()
0113 08:54:37.343509 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459407934 usec, L: 0
0113 08:54:37.366989 (+ 23480us) write_op.cc:270] Start()
0113 08:54:37.367033 (+ 44us) write_op.cc:276] Timestamp: P: 1736758459410704 usec, L: 0
0113 08:54:37.367272 (+ 239us) write_op.cc:270] Start()
0113 08:54:37.367297 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459412984 usec, L: 0
0113 08:54:37.367467 (+ 170us) write_op.cc:270] Start()
0113 08:54:37.367489 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459470793 usec, L: 0
0113 08:54:37.367689 (+ 200us) write_op.cc:270] Start()
0113 08:54:37.367712 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459494485 usec, L: 0
0113 08:54:37.413864 (+ 46152us) write_op.cc:270] Start()
0113 08:54:37.413918 (+ 54us) write_op.cc:276] Timestamp: P: 1736758459495256 usec, L: 0
0113 08:54:37.432195 (+ 18277us) write_op.cc:270] Start()
0113 08:54:37.432239 (+ 44us) write_op.cc:276] Timestamp: P: 1736758459514619 usec, L: 0
0113 08:54:37.435257 (+ 3018us) write_op.cc:270] Start()
0113 08:54:37.435288 (+ 31us) write_op.cc:276] Timestamp: P: 1736758459519621 usec, L: 0
0113 08:54:37.435478 (+ 190us) write_op.cc:270] Start()
0113 08:54:37.446015 (+ 10537us) write_op.cc:276] Timestamp: P: 1736758459545347 usec, L: 0
0113 08:54:37.453477 (+ 7462us) write_op.cc:270] Start()
0113 08:54:37.453522 (+ 45us) write_op.cc:276] Timestamp: P: 1736758459545925 usec, L: 0
0113 08:54:37.467764 (+ 14242us) write_op.cc:270] Start()
0113 08:54:37.467795 (+ 31us) write_op.cc:276] Timestamp: P: 1736758459573171 usec, L: 0
0113 08:54:37.468005 (+ 210us) write_op.cc:270] Start()
0113 08:54:37.468027 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459591046 usec, L: 0
0113 08:54:37.468178 (+ 151us) write_op.cc:270] Start()
0113 08:54:37.468197 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459591618 usec, L: 0
0113 08:54:37.468325 (+ 128us) write_op.cc:270] Start()
0113 08:54:37.468344 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459605179 usec, L: 0
0113 08:54:37.468505 (+ 161us) write_op.cc:270] Start()
0113 08:54:37.468523 (+ 18us) write_op.cc:276] Timestamp: P: 1736758459639644 usec, L: 0
0113 08:54:37.468698 (+ 175us) write_op.cc:270] Start()
0113 08:54:37.468719 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459649951 usec, L: 0
0113 08:54:37.468880 (+ 161us) write_op.cc:270] Start()
0113 08:54:37.468901 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459660708 usec, L: 0
0113 08:54:37.469171 (+ 270us) write_op.cc:270] Start()
0113 08:54:37.469191 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459662226 usec, L: 0
0113 08:54:37.469349 (+ 158us) write_op.cc:270] Start()
0113 08:54:37.469368 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459705958 usec, L: 0
0113 08:54:37.469521 (+ 153us) write_op.cc:270] Start()
0113 08:54:37.469541 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459738157 usec, L: 0
0113 08:54:37.469707 (+ 166us) write_op.cc:270] Start()
0113 08:54:37.469728 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459739274 usec, L: 0
0113 08:54:37.469853 (+ 125us) write_op.cc:270] Start()
0113 08:54:37.469870 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459744436 usec, L: 0
0113 08:54:37.482314 (+ 12444us) write_op.cc:270] Start()
0113 08:54:37.482345 (+ 31us) write_op.cc:276] Timestamp: P: 1736758459745156 usec, L: 0
0113 08:54:37.500381 (+ 18036us) write_op.cc:270] Start()
0113 08:54:37.500468 (+ 87us) write_op.cc:276] Timestamp: P: 1736758459791009 usec, L: 0
0113 08:54:37.504638 (+ 4170us) write_op.cc:270] Start()
0113 08:54:37.512217 (+ 7579us) write_op.cc:276] Timestamp: P: 1736758459795651 usec, L: 0
0113 08:54:37.514106 (+ 1889us) write_op.cc:270] Start()
0113 08:54:37.514133 (+ 27us) write_op.cc:276] Timestamp: P: 1736758459799771 usec, L: 0
0113 08:54:37.525954 (+ 11821us) write_op.cc:270] Start()
0113 08:54:37.525992 (+ 38us) write_op.cc:276] Timestamp: P: 1736758459804750 usec, L: 0
0113 08:54:37.527454 (+ 1462us) write_op.cc:270] Start()
0113 08:54:37.527485 (+ 31us) write_op.cc:276] Timestamp: P: 1736758459836004 usec, L: 0
0113 08:54:37.527674 (+ 189us) write_op.cc:270] Start()
0113 08:54:37.527693 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459843022 usec, L: 0
0113 08:54:37.527814 (+ 121us) write_op.cc:270] Start()
0113 08:54:37.527831 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459863833 usec, L: 0
0113 08:54:37.527946 (+ 115us) write_op.cc:270] Start()
0113 08:54:37.527962 (+ 16us) write_op.cc:276] Timestamp: P: 1736758459875852 usec, L: 0
0113 08:54:37.528076 (+ 114us) write_op.cc:270] Start()
0113 08:54:37.528093 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459913544 usec, L: 0
0113 08:54:37.528245 (+ 152us) write_op.cc:270] Start()
0113 08:54:37.528262 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459915597 usec, L: 0
0113 08:54:37.528386 (+ 124us) write_op.cc:270] Start()
0113 08:54:37.528403 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459934622 usec, L: 0
0113 08:54:37.528576 (+ 173us) write_op.cc:270] Start()
0113 08:54:37.528596 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459941786 usec, L: 0
0113 08:54:37.528727 (+ 131us) write_op.cc:270] Start()
0113 08:54:37.528745 (+ 18us) write_op.cc:276] Timestamp: P: 1736758459986958 usec, L: 0
0113 08:54:37.528912 (+ 167us) write_op.cc:270] Start()
0113 08:54:37.528933 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459991274 usec, L: 0
0113 08:54:37.529071 (+ 138us) write_op.cc:270] Start()
0113 08:54:37.529088 (+ 17us) write_op.cc:276] Timestamp: P: 1736758459991947 usec, L: 0
0113 08:54:37.529199 (+ 111us) write_op.cc:270] Start()
0113 08:54:37.529271 (+ 72us) write_op.cc:276] Timestamp: P: 1736758460013884 usec, L: 0
0113 08:54:37.529418 (+ 147us) write_op.cc:270] Start()
0113 08:54:37.529434 (+ 16us) write_op.cc:276] Timestamp: P: 1736758460034476 usec, L: 0
0113 08:54:37.529551 (+ 117us) write_op.cc:270] Start()
0113 08:54:37.529599 (+ 48us) write_op.cc:276] Timestamp: P: 1736758460040961 usec, L: 0
0113 08:54:37.564835 (+ 35236us) write_op.cc:270] Start()
0113 08:54:37.564859 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460063787 usec, L: 0
0113 08:54:37.565064 (+ 205us) write_op.cc:270] Start()
0113 08:54:37.565082 (+ 18us) write_op.cc:276] Timestamp: P: 1736758460066538 usec, L: 0
0113 08:54:37.565247 (+ 165us) write_op.cc:270] Start()
0113 08:54:37.565267 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460104560 usec, L: 0
0113 08:54:37.565439 (+ 172us) write_op.cc:270] Start()
0113 08:54:37.565454 (+ 15us) write_op.cc:276] Timestamp: P: 1736758460105330 usec, L: 0
0113 08:54:37.565554 (+ 100us) write_op.cc:270] Start()
0113 08:54:37.565568 (+ 14us) write_op.cc:276] Timestamp: P: 1736758460124800 usec, L: 0
0113 08:54:37.565658 (+ 90us) write_op.cc:270] Start()
0113 08:54:37.565670 (+ 12us) write_op.cc:276] Timestamp: P: 1736758460131733 usec, L: 0
0113 08:54:37.565765 (+ 95us) write_op.cc:270] Start()
0113 08:54:37.565778 (+ 13us) write_op.cc:276] Timestamp: P: 1736758460168531 usec, L: 0
0113 08:54:37.565903 (+ 125us) write_op.cc:270] Start()
0113 08:54:37.565924 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460176001 usec, L: 0
0113 08:54:37.566023 (+ 99us) write_op.cc:270] Start()
0113 08:54:37.566035 (+ 12us) write_op.cc:276] Timestamp: P: 1736758460185911 usec, L: 0
0113 08:54:37.566122 (+ 87us) write_op.cc:270] Start()
0113 08:54:37.566134 (+ 12us) write_op.cc:276] Timestamp: P: 1736758460187421 usec, L: 0
0113 08:54:37.566227 (+ 93us) write_op.cc:270] Start()
0113 08:54:37.566239 (+ 12us) write_op.cc:276] Timestamp: P: 1736758460219824 usec, L: 0
0113 08:54:37.566366 (+ 127us) write_op.cc:270] Start()
0113 08:54:37.566427 (+ 61us) write_op.cc:276] Timestamp: P: 1736758460230526 usec, L: 0
0113 08:54:37.566557 (+ 130us) write_op.cc:270] Start()
0113 08:54:37.566573 (+ 16us) write_op.cc:276] Timestamp: P: 1736758460252181 usec, L: 0
0113 08:54:37.566667 (+ 94us) write_op.cc:270] Start()
0113 08:54:37.566679 (+ 12us) write_op.cc:276] Timestamp: P: 1736758460254002 usec, L: 0
0113 08:54:37.566788 (+ 109us) write_op.cc:270] Start()
0113 08:54:37.566837 (+ 49us) write_op.cc:276] Timestamp: P: 1736758460286666 usec, L: 0
0113 08:54:37.566991 (+ 154us) write_op.cc:270] Start()
0113 08:54:37.567012 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460290500 usec, L: 0
0113 08:54:37.567158 (+ 146us) write_op.cc:270] Start()
0113 08:54:37.567178 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460301771 usec, L: 0
0113 08:54:37.595009 (+ 27831us) write_op.cc:270] Start()
0113 08:54:37.595032 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460339460 usec, L: 0
0113 08:54:37.595164 (+ 132us) write_op.cc:270] Start()
0113 08:54:37.595182 (+ 18us) write_op.cc:276] Timestamp: P: 1736758460344844 usec, L: 0
0113 08:54:37.595321 (+ 139us) write_op.cc:270] Start()
0113 08:54:37.595341 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460350615 usec, L: 0
0113 08:54:37.598616 (+ 3275us) write_op.cc:270] Start()
0113 08:54:37.599942 (+ 1326us) write_op.cc:276] Timestamp: P: 1736758460360308 usec, L: 0
0113 08:54:37.600211 (+ 269us) write_op.cc:270] Start()
0113 08:54:37.600248 (+ 37us) write_op.cc:276] Timestamp: P: 1736758460405866 usec, L: 0
0113 08:54:37.606838 (+ 6590us) write_op.cc:270] Start()
0113 08:54:37.606872 (+ 34us) write_op.cc:276] Timestamp: P: 1736758460415665 usec, L: 0
0113 08:54:37.611872 (+ 5000us) write_op.cc:270] Start()
0113 08:54:37.611904 (+ 32us) write_op.cc:276] Timestamp: P: 1736758460425095 usec, L: 0
0113 08:54:37.612144 (+ 240us) write_op.cc:270] Start()
0113 08:54:37.612170 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460427028 usec, L: 0
0113 08:54:37.628755 (+ 16585us) write_op.cc:270] Start()
0113 08:54:37.628788 (+ 33us) write_op.cc:276] Timestamp: P: 1736758460462086 usec, L: 0
0113 08:54:37.629033 (+ 245us) write_op.cc:270] Start()
0113 08:54:37.629058 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460486724 usec, L: 0
0113 08:54:37.646041 (+ 16983us) write_op.cc:270] Start()
0113 08:54:37.646076 (+ 35us) write_op.cc:276] Timestamp: P: 1736758460487496 usec, L: 0
0113 08:54:37.648476 (+ 2400us) write_op.cc:270] Start()
0113 08:54:37.648512 (+ 36us) write_op.cc:276] Timestamp: P: 1736758460505567 usec, L: 0
0113 08:54:37.655403 (+ 6891us) write_op.cc:270] Start()
0113 08:54:37.655438 (+ 35us) write_op.cc:276] Timestamp: P: 1736758460520069 usec, L: 0
0113 08:54:37.655688 (+ 250us) write_op.cc:270] Start()
0113 08:54:37.655713 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460569568 usec, L: 0
0113 08:54:37.655867 (+ 154us) write_op.cc:270] Start()
0113 08:54:37.655915 (+ 48us) write_op.cc:276] Timestamp: P: 1736758460574078 usec, L: 0
0113 08:54:37.656063 (+ 148us) write_op.cc:270] Start()
0113 08:54:37.656081 (+ 18us) write_op.cc:276] Timestamp: P: 1736758460575297 usec, L: 0
0113 08:54:37.656235 (+ 154us) write_op.cc:270] Start()
0113 08:54:37.656256 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460598657 usec, L: 0
0113 08:54:37.656425 (+ 169us) write_op.cc:270] Start()
0113 08:54:37.656448 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460611835 usec, L: 0
0113 08:54:37.656603 (+ 155us) write_op.cc:270] Start()
0113 08:54:37.656622 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460624053 usec, L: 0
0113 08:54:37.656768 (+ 146us) write_op.cc:270] Start()
0113 08:54:37.656787 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460648198 usec, L: 0
0113 08:54:37.656968 (+ 181us) write_op.cc:270] Start()
0113 08:54:37.656987 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460654228 usec, L: 0
0113 08:54:37.657166 (+ 179us) write_op.cc:270] Start()
0113 08:54:37.657187 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460673744 usec, L: 0
0113 08:54:37.715137 (+ 57950us) write_op.cc:270] Start()
0113 08:54:37.715168 (+ 31us) write_op.cc:276] Timestamp: P: 1736758460706418 usec, L: 0
0113 08:54:37.715374 (+ 206us) write_op.cc:270] Start()
0113 08:54:37.715399 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460707893 usec, L: 0
0113 08:54:37.717779 (+ 2380us) write_op.cc:270] Start()
0113 08:54:37.717811 (+ 32us) write_op.cc:276] Timestamp: P: 1736758460708495 usec, L: 0
0113 08:54:37.726852 (+ 9041us) write_op.cc:270] Start()
0113 08:54:37.726886 (+ 34us) write_op.cc:276] Timestamp: P: 1736758460732240 usec, L: 0
0113 08:54:37.732031 (+ 5145us) write_op.cc:270] Start()
0113 08:54:37.732058 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460764296 usec, L: 0
0113 08:54:37.732282 (+ 224us) write_op.cc:270] Start()
0113 08:54:37.732305 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460765878 usec, L: 0
0113 08:54:37.732457 (+ 152us) write_op.cc:270] Start()
0113 08:54:37.732480 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460775174 usec, L: 0
0113 08:54:37.762843 (+ 30363us) write_op.cc:270] Start()
0113 08:54:37.762881 (+ 38us) write_op.cc:276] Timestamp: P: 1736758460787525 usec, L: 0
0113 08:54:37.802410 (+ 39529us) write_op.cc:270] Start()
0113 08:54:37.802451 (+ 41us) write_op.cc:276] Timestamp: P: 1736758460828922 usec, L: 0
0113 08:54:37.805847 (+ 3396us) write_op.cc:270] Start()
0113 08:54:37.805869 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460832772 usec, L: 0
0113 08:54:37.806239 (+ 370us) write_op.cc:270] Start()
0113 08:54:37.806263 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460838508 usec, L: 0
0113 08:54:37.806474 (+ 211us) write_op.cc:270] Start()
0113 08:54:37.806500 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460867851 usec, L: 0
0113 08:54:37.806656 (+ 156us) write_op.cc:270] Start()
0113 08:54:37.806676 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460884346 usec, L: 0
0113 08:54:37.806864 (+ 188us) write_op.cc:270] Start()
0113 08:54:37.806897 (+ 33us) write_op.cc:276] Timestamp: P: 1736758460885648 usec, L: 0
0113 08:54:37.807054 (+ 157us) write_op.cc:270] Start()
0113 08:54:37.807073 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460905732 usec, L: 0
0113 08:54:37.807388 (+ 315us) write_op.cc:270] Start()
0113 08:54:37.807414 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460931060 usec, L: 0
0113 08:54:37.807577 (+ 163us) write_op.cc:270] Start()
0113 08:54:37.807598 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460949258 usec, L: 0
0113 08:54:37.807741 (+ 143us) write_op.cc:270] Start()
0113 08:54:37.807762 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460967624 usec, L: 0
0113 08:54:37.807924 (+ 162us) write_op.cc:270] Start()
0113 08:54:37.807967 (+ 43us) write_op.cc:276] Timestamp: P: 1736758460971580 usec, L: 0
0113 08:54:37.808157 (+ 190us) write_op.cc:270] Start()
0113 08:54:37.808179 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460987904 usec, L: 0
0113 08:54:37.808368 (+ 189us) write_op.cc:270] Start()
0113 08:54:37.808388 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461010131 usec, L: 0
0113 08:54:37.809166 (+ 778us) write_op.cc:270] Start()
0113 08:54:37.809194 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461013114 usec, L: 0
0113 08:54:37.809378 (+ 184us) write_op.cc:270] Start()
0113 08:54:37.809397 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461027401 usec, L: 0
0113 08:54:37.809587 (+ 190us) write_op.cc:270] Start()
0113 08:54:37.809608 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461065578 usec, L: 0
0113 08:54:37.809784 (+ 176us) write_op.cc:270] Start()
0113 08:54:37.809804 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461068959 usec, L: 0
0113 08:54:37.828341 (+ 18537us) write_op.cc:270] Start()
0113 08:54:37.828375 (+ 34us) write_op.cc:276] Timestamp: P: 1736758461089733 usec, L: 0
0113 08:54:37.834111 (+ 5736us) write_op.cc:270] Start()
0113 08:54:37.834150 (+ 39us) write_op.cc:276] Timestamp: P: 1736758461097530 usec, L: 0
0113 08:54:37.847473 (+ 13323us) write_op.cc:270] Start()
0113 08:54:37.847524 (+ 51us) write_op.cc:276] Timestamp: P: 1736758461123816 usec, L: 0
0113 08:54:37.847806 (+ 282us) write_op.cc:270] Start()
0113 08:54:37.847829 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461124542 usec, L: 0
0113 08:54:37.876711 (+ 28882us) write_op.cc:270] Start()
0113 08:54:37.876760 (+ 49us) write_op.cc:276] Timestamp: P: 1736758461175221 usec, L: 0
0113 08:54:37.880521 (+ 3761us) write_op.cc:270] Start()
0113 08:54:37.880570 (+ 49us) write_op.cc:276] Timestamp: P: 1736758461177554 usec, L: 0
0113 08:54:37.880784 (+ 214us) write_op.cc:270] Start()
0113 08:54:37.880803 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461180646 usec, L: 0
0113 08:54:37.883352 (+ 2549us) write_op.cc:270] Start()
0113 08:54:37.883383 (+ 31us) write_op.cc:276] Timestamp: P: 1736758461183765 usec, L: 0
0113 08:54:37.883595 (+ 212us) write_op.cc:270] Start()
0113 08:54:37.883619 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461220021 usec, L: 0
0113 08:54:37.883814 (+ 195us) write_op.cc:270] Start()
0113 08:54:37.883837 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461220742 usec, L: 0
0113 08:54:37.883985 (+ 148us) write_op.cc:270] Start()
0113 08:54:37.884004 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461231105 usec, L: 0
0113 08:54:37.892942 (+ 8938us) write_op.cc:270] Start()
0113 08:54:37.892971 (+ 29us) write_op.cc:276] Timestamp: P: 1736758461243508 usec, L: 0
0113 08:54:37.893146 (+ 175us) write_op.cc:270] Start()
0113 08:54:37.893171 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461289959 usec, L: 0
0113 08:54:37.893309 (+ 138us) write_op.cc:270] Start()
0113 08:54:37.893327 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461292498 usec, L: 0
0113 08:54:37.893485 (+ 158us) write_op.cc:270] Start()
0113 08:54:37.893502 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461295016 usec, L: 0
0113 08:54:37.894500 (+ 998us) write_op.cc:270] Start()
0113 08:54:37.894528 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461303192 usec, L: 0
0113 08:54:37.894707 (+ 179us) write_op.cc:270] Start()
0113 08:54:37.894726 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461340357 usec, L: 0
0113 08:54:37.894906 (+ 180us) write_op.cc:270] Start()
0113 08:54:37.894928 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461344105 usec, L: 0
0113 08:54:37.895090 (+ 162us) write_op.cc:270] Start()
0113 08:54:37.895129 (+ 39us) write_op.cc:276] Timestamp: P: 1736758461345366 usec, L: 0
0113 08:54:37.895265 (+ 136us) write_op.cc:270] Start()
0113 08:54:37.895283 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461375526 usec, L: 0
0113 08:54:37.895628 (+ 345us) write_op.cc:270] Start()
0113 08:54:37.895661 (+ 33us) write_op.cc:276] Timestamp: P: 1736758461394671 usec, L: 0
0113 08:54:37.895812 (+ 151us) write_op.cc:270] Start()
0113 08:54:37.897511 (+ 1699us) write_op.cc:276] Timestamp: P: 1736758461396749 usec, L: 0
0113 08:54:37.899168 (+ 1657us) write_op.cc:270] Start()
0113 08:54:37.899198 (+ 30us) write_op.cc:276] Timestamp: P: 1736758461411938 usec, L: 0
0113 08:54:37.899395 (+ 197us) write_op.cc:270] Start()
0113 08:54:37.899423 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461441698 usec, L: 0
0113 08:54:37.899598 (+ 175us) write_op.cc:270] Start()
0113 08:54:37.899619 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461474301 usec, L: 0
0113 08:54:37.900473 (+ 854us) write_op.cc:270] Start()
0113 08:54:37.900502 (+ 29us) write_op.cc:276] Timestamp: P: 1736758461493029 usec, L: 0
0113 08:54:37.900705 (+ 203us) write_op.cc:270] Start()
0113 08:54:37.900728 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461493814 usec, L: 0
0113 08:54:37.917540 (+ 16812us) write_op.cc:270] Start()
0113 08:54:37.917575 (+ 35us) write_op.cc:276] Timestamp: P: 1736758461497574 usec, L: 0
0113 08:54:37.920422 (+ 2847us) write_op.cc:270] Start()
0113 08:54:37.920470 (+ 48us) write_op.cc:276] Timestamp: P: 1736758461543730 usec, L: 0
0113 08:54:37.921983 (+ 1513us) write_op.cc:270] Start()
0113 08:54:37.922011 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461548126 usec, L: 0
0113 08:54:37.924343 (+ 2332us) write_op.cc:270] Start()
0113 08:54:37.924374 (+ 31us) write_op.cc:276] Timestamp: P: 1736758461568853 usec, L: 0
0113 08:54:37.939817 (+ 15443us) write_op.cc:270] Start()
0113 08:54:37.939850 (+ 33us) write_op.cc:276] Timestamp: P: 1736758461579114 usec, L: 0
0113 08:54:37.949804 (+ 9954us) write_op.cc:270] Start()
0113 08:54:37.949839 (+ 35us) write_op.cc:276] Timestamp: P: 1736758461601726 usec, L: 0
0113 08:54:37.963560 (+ 13721us) write_op.cc:270] Start()
0113 08:54:37.963597 (+ 37us) write_op.cc:276] Timestamp: P: 1736758461643477 usec, L: 0
0113 08:54:37.963826 (+ 229us) write_op.cc:270] Start()
0113 08:54:37.963858 (+ 32us) write_op.cc:276] Timestamp: P: 1736758461647443 usec, L: 0
0113 08:54:37.964012 (+ 154us) write_op.cc:270] Start()
0113 08:54:37.964032 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461653374 usec, L: 0
0113 08:54:37.964195 (+ 163us) write_op.cc:270] Start()
0113 08:54:37.964221 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461673491 usec, L: 0
0113 08:54:37.971040 (+ 6819us) write_op.cc:270] Start()
0113 08:54:37.971068 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461706884 usec, L: 0
0113 08:54:37.973235 (+ 2167us) write_op.cc:270] Start()
0113 08:54:37.973274 (+ 39us) write_op.cc:276] Timestamp: P: 1736758461712019 usec, L: 0
0113 08:54:37.977960 (+ 4686us) write_op.cc:270] Start()
0113 08:54:37.978007 (+ 47us) write_op.cc:276] Timestamp: P: 1736758461714710 usec, L: 0
0113 08:54:37.978249 (+ 242us) write_op.cc:270] Start()
0113 08:54:37.978269 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461747812 usec, L: 0
0113 08:54:37.981442 (+ 3173us) write_op.cc:270] Start()
0113 08:54:37.981464 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461761247 usec, L: 0
0113 08:54:37.987020 (+ 5556us) write_op.cc:270] Start()
0113 08:54:37.987050 (+ 30us) write_op.cc:276] Timestamp: P: 1736758461781552 usec, L: 0
0113 08:54:37.987235 (+ 185us) write_op.cc:270] Start()
0113 08:54:37.987256 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461786660 usec, L: 0
0113 08:54:37.998090 (+ 10834us) write_op.cc:270] Start()
0113 08:54:37.998125 (+ 35us) write_op.cc:276] Timestamp: P: 1736758461832485 usec, L: 0
0113 08:54:38.001834 (+ 3709us) write_op.cc:270] Start()
0113 08:54:38.001866 (+ 32us) write_op.cc:276] Timestamp: P: 1736758461853877 usec, L: 0
0113 08:54:38.021669 (+ 19803us) write_op.cc:270] Start()
0113 08:54:38.021721 (+ 52us) write_op.cc:276] Timestamp: P: 1736758461892572 usec, L: 0
0113 08:54:38.045412 (+ 23691us) write_op.cc:270] Start()
0113 08:54:38.045455 (+ 43us) write_op.cc:276] Timestamp: P: 1736758461914493 usec, L: 0
0113 08:54:38.057294 (+ 11839us) write_op.cc:270] Start()
0113 08:54:38.057329 (+ 35us) write_op.cc:276] Timestamp: P: 1736758461928284 usec, L: 0
0113 08:54:38.059980 (+ 2651us) write_op.cc:270] Start()
0113 08:54:38.060016 (+ 36us) write_op.cc:2
I20250113 08:54:46.315632 8777 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:46.315768 8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.3:55187 (request call id 534) took 1609 ms. Trace:
I20250113 08:54:46.315843 8777 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:46.418650 8777 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:46.421306 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 1395 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:46.422524 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 1395 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:46.435029 8581 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:46.437537 8777 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250113 08:54:46.654675 7905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.123.3:46625, user_credentials={real_user=slave}} blocked reactor thread for 43969.6us
I20250113 08:54:46.823879 8777 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:46.824088 8777 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:46.824801 8777 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:46.832753 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 1395 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:46.860776 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 1395 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:46.865633 8581 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:46.865975 8777 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250113 08:54:47.030951 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
I20250113 08:54:47.360668 8777 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:47.360873 8777 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:47.361606 8777 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:47.364461 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 1395 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:47.368021 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 1395 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:47.378481 8581 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:47.383487 8777 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:46.315876 8656 rpcz_store.cc:276] 0113 08:54:44.706157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:44.706235 (+ 78us) service_pool.cc:225] Handling call
0113 08:54:44.802695 (+ 96460us) raft_consensus.cc:1466] Updating replica for 425 ops
0113 08:54:45.007274 (+204579us) raft_consensus.cc:1529] Early marking committed up to index 970
0113 08:54:45.007388 (+ 114us) raft_consensus.cc:1534] Triggering prepare for 425 ops
0113 08:54:45.007547 (+ 159us) write_op.cc:270] Start()
0113 08:54:45.007578 (+ 31us) write_op.cc:276] Timestamp: P: 1736758475620554 usec, L: 0
0113 08:54:45.010754 (+ 3176us) write_op.cc:270] Start()
0113 08:54:45.010795 (+ 41us) write_op.cc:276] Timestamp: P: 1736758475630400 usec, L: 0
0113 08:54:45.011063 (+ 268us) write_op.cc:270] Start()
0113 08:54:45.011084 (+ 21us) write_op.cc:276] Timestamp: P: 1736758475676399 usec, L: 0
0113 08:54:45.011752 (+ 668us) write_op.cc:270] Start()
0113 08:54:45.011777 (+ 25us) write_op.cc:276] Timestamp: P: 1736758475679514 usec, L: 0
0113 08:54:45.011954 (+ 177us) write_op.cc:270] Start()
0113 08:54:45.011976 (+ 22us) write_op.cc:276] Timestamp: P: 1736758475692787 usec, L: 0
0113 08:54:45.012115 (+ 139us) write_op.cc:270] Start()
0113 08:54:45.012135 (+ 20us) write_op.cc:276] Timestamp: P: 1736758475710701 usec, L: 0
0113 08:54:45.012285 (+ 150us) write_op.cc:270] Start()
0113 08:54:45.012329 (+ 44us) write_op.cc:276] Timestamp: P: 1736758475743523 usec, L: 0
0113 08:54:45.012502 (+ 173us) write_op.cc:270] Start()
0113 08:54:45.012524 (+ 22us) write_op.cc:276] Timestamp: P: 1736758475744697 usec, L: 0
0113 08:54:45.012670 (+ 146us) write_op.cc:270] Start()
0113 08:54:45.012691 (+ 21us) write_op.cc:276] Timestamp: P: 1736758475747807 usec, L: 0
0113 08:54:45.012834 (+ 143us) write_op.cc:270] Start()
0113 08:54:45.012852 (+ 18us) write_op.cc:276] Timestamp: P: 1736758475781056 usec, L: 0
0113 08:54:45.013029 (+ 177us) write_op.cc:270] Start()
0113 08:54:45.013049 (+ 20us) write_op.cc:276] Timestamp: P: 1736758475812800 usec, L: 0
0113 08:54:45.013204 (+ 155us) write_op.cc:270] Start()
0113 08:54:45.013223 (+ 19us) write_op.cc:276] Timestamp: P: 1736758475814512 usec, L: 0
0113 08:54:45.013364 (+ 141us) write_op.cc:270] Start()
0113 08:54:45.013384 (+ 20us) write_op.cc:276] Timestamp: P: 1736758475816366 usec, L: 0
0113 08:54:45.013518 (+ 134us) write_op.cc:270] Start()
0113 08:54:45.013538 (+ 20us) write_op.cc:276] Timestamp: P: 1736758475860083 usec, L: 0
0113 08:54:45.013671 (+ 133us) write_op.cc:270] Start()
0113 08:54:45.013691 (+ 20us) write_op.cc:276] Timestamp: P: 1736758475884480 usec, L: 0
0113 08:54:45.013827 (+ 136us) write_op.cc:270] Start()
0113 08:54:45.013846 (+ 19us) write_op.cc:276] Timestamp: P: 1736758475888912 usec, L: 0
0113 08:54:45.014051 (+ 205us) write_op.cc:270] Start()
0113 08:54:45.014075 (+ 24us) write_op.cc:276] Timestamp: P: 1736758475891407 usec, L: 0
0113 08:54:45.014229 (+ 154us) write_op.cc:270] Start()
0113 08:54:45.014251 (+ 22us) write_op.cc:276] Timestamp: P: 1736758475903418 usec, L: 0
0113 08:54:45.014388 (+ 137us) write_op.cc:270] Start()
0113 08:54:45.014407 (+ 19us) write_op.cc:276] Timestamp: P: 1736758475944889 usec, L: 0
0113 08:54:45.014543 (+ 136us) write_op.cc:270] Start()
0113 08:54:45.014575 (+ 32us) write_op.cc:276] Timestamp: P: 1736758475955187 usec, L: 0
0113 08:54:45.014747 (+ 172us) write_op.cc:270] Start()
0113 08:54:45.014767 (+ 20us) write_op.cc:276] Timestamp: P: 1736758475958208 usec, L: 0
0113 08:54:45.014972 (+ 205us) write_op.cc:270] Start()
0113 08:54:45.014997 (+ 25us) write_op.cc:276] Timestamp: P: 1736758475971253 usec, L: 0
0113 08:54:45.015985 (+ 988us) write_op.cc:270] Start()
0113 08:54:45.016011 (+ 26us) write_op.cc:276] Timestamp: P: 1736758476012368 usec, L: 0
0113 08:54:45.017714 (+ 1703us) write_op.cc:270] Start()
0113 08:54:45.017743 (+ 29us) write_op.cc:276] Timestamp: P: 1736758476037551 usec, L: 0
0113 08:54:45.018135 (+ 392us) write_op.cc:270] Start()
0113 08:54:45.018158 (+ 23us) write_op.cc:276] Timestamp: P: 1736758476059314 usec, L: 0
0113 08:54:45.018581 (+ 423us) write_op.cc:270] Start()
0113 08:54:45.018604 (+ 23us) write_op.cc:276] Timestamp: P: 1736758476060661 usec, L: 0
0113 08:54:45.019017 (+ 413us) write_op.cc:270] Start()
0113 08:54:45.019039 (+ 22us) write_op.cc:276] Timestamp: P: 1736758476096483 usec, L: 0
0113 08:54:45.019180 (+ 141us) write_op.cc:270] Start()
0113 08:54:45.019202 (+ 22us) write_op.cc:276] Timestamp: P: 1736758476099060 usec, L: 0
0113 08:54:45.019338 (+ 136us) write_op.cc:270] Start()
0113 08:54:45.019357 (+ 19us) write_op.cc:276] Timestamp: P: 1736758476139869 usec, L: 0
0113 08:54:45.019494 (+ 137us) write_op.cc:270] Start()
0113 08:54:45.019514 (+ 20us) write_op.cc:276] Timestamp: P: 1736758476174713 usec, L: 0
0113 08:54:45.021694 (+ 2180us) write_op.cc:270] Start()
0113 08:54:45.021732 (+ 38us) write_op.cc:276] Timestamp: P: 1736758476182564 usec, L: 0
0113 08:54:45.023488 (+ 1756us) write_op.cc:270] Start()
0113 08:54:45.023515 (+ 27us) write_op.cc:276] Timestamp: P: 1736758476187855 usec, L: 0
0113 08:54:45.098910 (+ 75395us) write_op.cc:270] Start()
0113 08:54:45.098964 (+ 54us) write_op.cc:276] Timestamp: P: 1736758476235608 usec, L: 0
0113 08:54:45.099296 (+ 332us) write_op.cc:270] Start()
0113 08:54:45.099316 (+ 20us) write_op.cc:276] Timestamp: P: 1736758476282746 usec, L: 0
0113 08:54:45.099488 (+ 172us) write_op.cc:270] Start()
0113 08:54:45.099507 (+ 19us) write_op.cc:276] Timestamp: P: 1736758476284715 usec, L: 0
0113 08:54:45.099662 (+ 155us) write_op.cc:270] Start()
0113 08:54:45.099679 (+ 17us) write_op.cc:276] Timestamp: P: 1736758476285290 usec, L: 0
0113 08:54:45.099829 (+ 150us) write_op.cc:270] Start()
0113 08:54:45.099848 (+ 19us) write_op.cc:276] Timestamp: P: 1736758476311301 usec, L: 0
0113 08:54:45.100038 (+ 190us) write_op.cc:270] Start()
0113 08:54:45.100057 (+ 19us) write_op.cc:276] Timestamp: P: 1736758476341594 usec, L: 0
0113 08:54:45.100245 (+ 188us) write_op.cc:270] Start()
0113 08:54:45.100266 (+ 21us) write_op.cc:276] Timestamp: P: 1736758476362446 usec, L: 0
0113 08:54:45.100484 (+ 218us) write_op.cc:270] Start()
0113 08:54:45.100508 (+ 24us) write_op.cc:276] Timestamp: P: 1736758476366313 usec, L: 0
0113 08:54:45.100672 (+ 164us) write_op.cc:270] Start()
0113 08:54:45.100690 (+ 18us) write_op.cc:276] Timestamp: P: 1736758476407670 usec, L: 0
0113 08:54:45.100894 (+ 204us) write_op.cc:270] Start()
0113 08:54:45.100925 (+ 31us) write_op.cc:276] Timestamp: P: 1736758476445620 usec, L: 0
0113 08:54:45.101102 (+ 177us) write_op.cc:270] Start()
0113 08:54:45.101121 (+ 19us) write_op.cc:276] Timestamp: P: 1736758476447128 usec, L: 0
0113 08:54:45.101277 (+ 156us) write_op.cc:270] Start()
0113 08:54:45.101296 (+ 19us) write_op.cc:276] Timestamp: P: 1736758476447773 usec, L: 0
0113 08:54:45.101505 (+ 209us) write_op.cc:270] Start()
0113 08:54:45.101525 (+ 20us) write_op.cc:276] Timestamp: P: 1736758476493112 usec, L: 0
0113 08:54:45.101684 (+ 159us) write_op.cc:270] Start()
0113 08:54:45.101702 (+ 18us) write_op.cc:276] Timestamp: P: 1736758476521267 usec, L: 0
0113 08:54:45.101890 (+ 188us) write_op.cc:270] Start()
0113 08:54:45.101921 (+ 31us) write_op.cc:276] Timestamp: P: 1736758476522017 usec, L: 0
0113 08:54:45.102103 (+ 182us) write_op.cc:270] Start()
0113 08:54:45.102123 (+ 20us) write_op.cc:276] Timestamp: P: 1736758476524220 usec, L: 0
0113 08:54:45.102282 (+ 159us) write_op.cc:270] Start()
0113 08:54:45.102302 (+ 20us) write_op.cc:276] Timestamp: P: 1736758476624049 usec, L: 0
0113 08:54:45.102459 (+ 157us) write_op.cc:270] Start()
0113 08:54:45.102479 (+ 20us) write_op.cc:276] Timestamp: P: 1736758476625786 usec, L: 0
0113 08:54:45.102744 (+ 265us) write_op.cc:270] Start()
0113 08:54:45.102766 (+ 22us) write_op.cc:276] Timestamp: P: 1736758476626656 usec, L: 0
0113 08:54:45.113522 (+ 10756us) write_op.cc:270] Start()
0113 08:54:45.113555 (+ 33us) write_op.cc:276] Timestamp: P: 1736758476629576 usec, L: 0
0113 08:54:45.114996 (+ 1441us) write_op.cc:270] Start()
0113 08:54:45.115020 (+ 24us) write_op.cc:276] Timestamp: P: 1736758476684338 usec, L: 0
0113 08:54:45.115167 (+ 147us) write_op.cc:270] Start()
0113 08:54:45.115186 (+ 19us) write_op.cc:276] Timestamp: P: 1736758476708342 usec, L: 0
0113 08:54:45.115337 (+ 151us) write_op.cc:270] Start()
0113 08:54:45.115358 (+ 21us) write_op.cc:276] Timestamp: P: 1736758476708939 usec, L: 0
0113 08:54:45.121160 (+ 5802us) write_op.cc:270] Start()
0113 08:54:45.121186 (+ 26us) write_op.cc:276] Timestamp: P: 1736758476709507 usec, L: 0
0113 08:54:45.123617 (+ 2431us) write_op.cc:270] Start()
0113 08:54:45.123876 (+ 259us) write_op.cc:276] Timestamp: P: 1736758476759448 usec, L: 0
0113 08:54:45.125089 (+ 1213us) write_op.cc:270] Start()
0113 08:54:45.125183 (+ 94us) write_op.cc:276] Timestamp: P: 1736758476780456 usec, L: 0
0113 08:54:45.127510 (+ 2327us) write_op.cc:270] Start()
0113 08:54:45.127535 (+ 25us) write_op.cc:276] Timestamp: P: 1736758476805607 usec, L: 0
0113 08:54:45.139808 (+ 12273us) write_op.cc:270] Start()
0113 08:54:45.139864 (+ 56us) write_op.cc:276] Timestamp: P: 1736758476810207 usec, L: 0
0113 08:54:45.140144 (+ 280us) write_op.cc:270] Start()
0113 08:54:45.140187 (+ 43us) write_op.cc:276] Timestamp: P: 1736758476812718 usec, L: 0
0113 08:54:45.140389 (+ 202us) write_op.cc:270] Start()
0113 08:54:45.140409 (+ 20us) write_op.cc:276] Timestamp: P: 1736758476882598 usec, L: 0
0113 08:54:45.140587 (+ 178us) write_op.cc:270] Start()
0113 08:54:45.140608 (+ 21us) write_op.cc:276] Timestamp: P: 1736758476884887 usec, L: 0
0113 08:54:45.140779 (+ 171us) write_op.cc:270] Start()
0113 08:54:45.140799 (+ 20us) write_op.cc:276] Timestamp: P: 1736758476886216 usec, L: 0
0113 08:54:45.141023 (+ 224us) write_op.cc:270] Start()
0113 08:54:45.141044 (+ 21us) write_op.cc:276] Timestamp: P: 1736758476892219 usec, L: 0
0113 08:54:45.141322 (+ 278us) write_op.cc:270] Start()
0113 08:54:45.141345 (+ 23us) write_op.cc:276] Timestamp: P: 1736758476958419 usec, L: 0
0113 08:54:45.141512 (+ 167us) write_op.cc:270] Start()
0113 08:54:45.141538 (+ 26us) write_op.cc:276] Timestamp: P: 1736758476962427 usec, L: 0
0113 08:54:45.141742 (+ 204us) write_op.cc:270] Start()
0113 08:54:45.141766 (+ 24us) write_op.cc:276] Timestamp: P: 1736758476966089 usec, L: 0
0113 08:54:45.141964 (+ 198us) write_op.cc:270] Start()
0113 08:54:45.141999 (+ 35us) write_op.cc:276] Timestamp: P: 1736758476972452 usec, L: 0
0113 08:54:45.142200 (+ 201us) write_op.cc:270] Start()
0113 08:54:45.142223 (+ 23us) write_op.cc:276] Timestamp: P: 1736758477034210 usec, L: 0
0113 08:54:45.142438 (+ 215us) write_op.cc:270] Start()
0113 08:54:45.142460 (+ 22us) write_op.cc:276] Timestamp: P: 1736758477038504 usec, L: 0
0113 08:54:45.142709 (+ 249us) write_op.cc:270] Start()
0113 08:54:45.142734 (+ 25us) write_op.cc:276] Timestamp: P: 1736758477053677 usec, L: 0
0113 08:54:45.142962 (+ 228us) write_op.cc:270] Start()
0113 08:54:45.142986 (+ 24us) write_op.cc:276] Timestamp: P: 1736758477055801 usec, L: 0
0113 08:54:45.159967 (+ 16981us) write_op.cc:270] Start()
0113 08:54:45.159996 (+ 29us) write_op.cc:276] Timestamp: P: 1736758477112846 usec, L: 0
0113 08:54:45.171005 (+ 11009us) write_op.cc:270] Start()
0113 08:54:45.171029 (+ 24us) write_op.cc:276] Timestamp: P: 1736758477113709 usec, L: 0
0113 08:54:45.171266 (+ 237us) write_op.cc:270] Start()
0113 08:54:45.171288 (+ 22us) write_op.cc:276] Timestamp: P: 1736758477153938 usec, L: 0
0113 08:54:45.171477 (+ 189us) write_op.cc:270] Start()
0113 08:54:45.171498 (+ 21us) write_op.cc:276] Timestamp: P: 1736758477154685 usec, L: 0
0113 08:54:45.171678 (+ 180us) write_op.cc:270] Start()
0113 08:54:45.171699 (+ 21us) write_op.cc:276] Timestamp: P: 1736758477182866 usec, L: 0
0113 08:54:45.171863 (+ 164us) write_op.cc:270] Start()
0113 08:54:45.171884 (+ 21us) write_op.cc:276] Timestamp: P: 1736758477183631 usec, L: 0
0113 08:54:45.172085 (+ 201us) write_op.cc:270] Start()
0113 08:54:45.172108 (+ 23us) write_op.cc:276] Timestamp: P: 1736758477216082 usec, L: 0
0113 08:54:45.172293 (+ 185us) write_op.cc:270] Start()
0113 08:54:45.172316 (+ 23us) write_op.cc:276] Timestamp: P: 1736758477234418 usec, L: 0
0113 08:54:45.172489 (+ 173us) write_op.cc:270] Start()
0113 08:54:45.172509 (+ 20us) write_op.cc:276] Timestamp: P: 1736758477266836 usec, L: 0
0113 08:54:45.172673 (+ 164us) write_op.cc:270] Start()
0113 08:54:45.172693 (+ 20us) write_op.cc:276] Timestamp: P: 1736758477267462 usec, L: 0
0113 08:54:45.172843 (+ 150us) write_op.cc:270] Start()
0113 08:54:45.172862 (+ 19us) write_op.cc:276] Timestamp: P: 1736758477281526 usec, L: 0
0113 08:54:45.183069 (+ 10207us) write_op.cc:270] Start()
0113 08:54:45.183119 (+ 50us) write_op.cc:276] Timestamp: P: 1736758477297119 usec, L: 0
0113 08:54:45.184252 (+ 1133us) write_op.cc:270] Start()
0113 08:54:45.184280 (+ 28us) write_op.cc:276] Timestamp: P: 1736758477308428 usec, L: 0
0113 08:54:45.184499 (+ 219us) write_op.cc:270] Start()
0113 08:54:45.184520 (+ 21us) write_op.cc:276] Timestamp: P: 1736758477348453 usec, L: 0
0113 08:54:45.185509 (+ 989us) write_op.cc:270] Start()
0113 08:54:45.185538 (+ 29us) write_op.cc:276] Timestamp: P: 1736758477367685 usec, L: 0
0113 08:54:45.185755 (+ 217us) write_op.cc:270] Start()
0113 08:54:45.185780 (+ 25us) write_op.cc:276] Timestamp: P: 1736758477369078 usec, L: 0
0113 08:54:45.187343 (+ 1563us) write_op.cc:270] Start()
0113 08:54:45.187373 (+ 30us) write_op.cc:276] Timestamp: P: 1736758477390072 usec, L: 0
0113 08:54:45.187596 (+ 223us) write_op.cc:270] Start()
0113 08:54:45.187628 (+ 32us) write_op.cc:276] Timestamp: P: 1736758477409392 usec, L: 0
0113 08:54:45.187962 (+ 334us) write_op.cc:270] Start()
0113 08:54:45.189609 (+ 1647us) write_op.cc:276] Timestamp: P: 1736758477460443 usec, L: 0
0113 08:54:45.189849 (+ 240us) write_op.cc:270] Start()
0113 08:54:45.189876 (+ 27us) write_op.cc:276] Timestamp: P: 1736758477462148 usec, L: 0
0113 08:54:45.202214 (+ 12338us) write_op.cc:270] Start()
0113 08:54:45.202243 (+ 29us) write_op.cc:276] Timestamp: P: 1736758477475114 usec, L: 0
0113 08:54:45.202684 (+ 441us) write_op.cc:270] Start()
0113 08:54:45.202711 (+ 27us) write_op.cc:276] Timestamp: P: 1736758477487178 usec, L: 0
0113 08:54:45.213644 (+ 10933us) write_op.cc:270] Start()
0113 08:54:45.213702 (+ 58us) write_op.cc:276] Timestamp: P: 1736758477507087 usec, L: 0
0113 08:54:45.214453 (+ 751us) write_op.cc:270] Start()
0113 08:54:45.214487 (+ 34us) write_op.cc:276] Timestamp: P: 1736758477568731 usec, L: 0
0113 08:54:45.214654 (+ 167us) write_op.cc:270] Start()
0113 08:54:45.214674 (+ 20us) write_op.cc:276] Timestamp: P: 1736758477569343 usec, L: 0
0113 08:54:45.214856 (+ 182us) write_op.cc:270] Start()
0113 08:54:45.214884 (+ 28us) write_op.cc:276] Timestamp: P: 1736758477605496 usec, L: 0
0113 08:54:45.215057 (+ 173us) write_op.cc:270] Start()
0113 08:54:45.215079 (+ 22us) write_op.cc:276] Timestamp: P: 1736758477608757 usec, L: 0
0113 08:54:45.215269 (+ 190us) write_op.cc:270] Start()
0113 08:54:45.215294 (+ 25us) write_op.cc:276] Timestamp: P: 1736758477652975 usec, L: 0
0113 08:54:45.215472 (+ 178us) write_op.cc:270] Start()
0113 08:54:45.215509 (+ 37us) write_op.cc:276] Timestamp: P: 1736758477684191 usec, L: 0
0113 08:54:45.215714 (+ 205us) write_op.cc:270] Start()
0113 08:54:45.215735 (+ 21us) write_op.cc:276] Timestamp: P: 1736758477706363 usec, L: 0
0113 08:54:45.215922 (+ 187us) write_op.cc:270] Start()
0113 08:54:45.215944 (+ 22us) write_op.cc:276] Timestamp: P: 1736758477707060 usec, L: 0
0113 08:54:45.216129 (+ 185us) write_op.cc:270] Start()
0113 08:54:45.216151 (+ 22us) write_op.cc:276] Timestamp: P: 1736758477723715 usec, L: 0
0113 08:54:45.216360 (+ 209us) write_op.cc:270] Start()
0113 08:54:45.216382 (+ 22us) write_op.cc:276] Timestamp: P: 1736758477768125 usec, L: 0
0113 08:54:45.218041 (+ 1659us) write_op.cc:270] Start()
0113 08:54:45.218064 (+ 23us) write_op.cc:276] Timestamp: P: 1736758477784865 usec, L: 0
0113 08:54:45.218591 (+ 527us) write_op.cc:270] Start()
0113 08:54:45.218613 (+ 22us) write_op.cc:276] Timestamp: P: 1736758477795575 usec, L: 0
0113 08:54:45.224997 (+ 6384us) write_op.cc:270] Start()
0113 08:54:45.225019 (+ 22us) write_op.cc:276] Timestamp: P: 1736758477816000 usec, L: 0
0113 08:54:45.237389 (+ 12370us) write_op.cc:270] Start()
0113 08:54:45.237418 (+ 29us) write_op.cc:276] Timestamp: P: 1736758477816649 usec, L: 0
0113 08:54:45.237587 (+ 169us) write_op.cc:270] Start()
0113 08:54:45.237610 (+ 23us) write_op.cc:276] Timestamp: P: 1736758477866834 usec, L: 0
0113 08:54:45.237764 (+ 154us) write_op.cc:270] Start()
0113 08:54:45.237787 (+ 23us) write_op.cc:276] Timestamp: P: 1736758477869400 usec, L: 0
0113 08:54:45.237928 (+ 141us) write_op.cc:270] Start()
0113 08:54:45.237950 (+ 22us) write_op.cc:276] Timestamp: P: 1736758477884456 usec, L: 0
0113 08:54:45.238095 (+ 145us) write_op.cc:270] Start()
0113 08:54:45.238114 (+ 19us) write_op.cc:276] Timestamp: P: 1736758477891049 usec, L: 0
0113 08:54:45.242619 (+ 4505us) write_op.cc:270] Start()
0113 08:54:45.242647 (+ 28us) write_op.cc:276] Timestamp: P: 1736758477963018 usec, L: 0
0113 08:54:45.242895 (+ 248us) write_op.cc:270] Start()
0113 08:54:45.242920 (+ 25us) write_op.cc:276] Timestamp: P: 1736758477970980 usec, L: 0
0113 08:54:45.243623 (+ 703us) write_op.cc:270] Start()
0113 08:54:45.243650 (+ 27us) write_op.cc:276] Timestamp: P: 1736758477985369 usec, L: 0
0113 08:54:45.244815 (+ 1165us) write_op.cc:270] Start()
0113 08:54:45.244840 (+ 25us) write_op.cc:276] Timestamp: P: 1736758477995037 usec, L: 0
0113 08:54:45.246765 (+ 1925us) write_op.cc:270] Start()
0113 08:54:45.246795 (+ 30us) write_op.cc:276] Timestamp: P: 1736758478053696 usec, L: 0
0113 08:54:45.247028 (+ 233us) write_op.cc:270] Start()
0113 08:54:45.247058 (+ 30us) write_op.cc:276] Timestamp: P: 1736758478059778 usec, L: 0
0113 08:54:45.251018 (+ 3960us) write_op.cc:270] Start()
0113 08:54:45.251042 (+ 24us) write_op.cc:276] Timestamp: P: 1736758478071525 usec, L: 0
0113 08:54:45.253628 (+ 2586us) write_op.cc:270] Start()
0113 08:54:45.253657 (+ 29us) write_op.cc:276] Timestamp: P: 1736758478072813 usec, L: 0
0113 08:54:45.253819 (+ 162us) write_op.cc:270] Start()
0113 08:54:45.253839 (+ 20us) write_op.cc:276] Timestamp: P: 1736758478133222 usec, L: 0
0113 08:54:45.254011 (+ 172us) write_op.cc:270] Start()
0113 08:54:45.254031 (+ 20us) write_op.cc:276] Timestamp: P: 1736758478133906 usec, L: 0
0113 08:54:45.254152 (+ 121us) write_op.cc:270] Start()
0113 08:54:45.254172 (+ 20us) write_op.cc:276] Timestamp: P: 1736758478139224 usec, L: 0
0113 08:54:45.254312 (+ 140us) write_op.cc:270] Start()
0113 08:54:45.254329 (+ 17us) write_op.cc:276] Timestamp: P: 1736758478142600 usec, L: 0
0113 08:54:45.254451 (+ 122us) write_op.cc:270] Start()
0113 08:54:45.254468 (+ 17us) write_op.cc:276] Timestamp: P: 1736758478207018 usec, L: 0
0113 08:54:45.254590 (+ 122us) write_op.cc:270] Start()
0113 08:54:45.254607 (+ 17us) write_op.cc:276] Timestamp: P: 1736758478211509 usec, L: 0
0113 08:54:45.254763 (+ 156us) write_op.cc:270] Start()
0113 08:54:45.254780 (+ 17us) write_op.cc:276] Timestamp: P: 1736758478213391 usec, L: 0
0113 08:54:45.254972 (+ 192us) write_op.cc:270] Start()
0113 08:54:45.254992 (+ 20us) write_op.cc:276] Timestamp: P: 1736758478233051 usec, L: 0
0113 08:54:45.258756 (+ 3764us) write_op.cc:270] Start()
0113 08:54:45.258839 (+ 83us) write_op.cc:276] Timestamp: P: 1736758478264165 usec, L: 0
0113 08:54:45.259106 (+ 267us) write_op.cc:270] Start()
0113 08:54:45.259130 (+ 24us) write_op.cc:276] Timestamp: P: 1736758478286870 usec, L: 0
0113 08:54:45.260690 (+ 1560us) write_op.cc:270] Start()
0113 08:54:45.260715 (+ 25us) write_op.cc:276] Timestamp: P: 1736758478311353 usec, L: 0
0113 08:54:45.260933 (+ 218us) write_op.cc:270] Start()
0113 08:54:45.260956 (+ 23us) write_op.cc:276] Timestamp: P: 1736758478313475 usec, L: 0
0113 08:54:45.261168 (+ 212us) write_op.cc:270] Start()
0113 08:54:45.261202 (+ 34us) write_op.cc:276] Timestamp: P: 1736758478348409 usec, L: 0
0113 08:54:45.262902 (+ 1700us) write_op.cc:270] Start()
0113 08:54:45.262932 (+ 30us) write_op.cc:276] Timestamp: P: 1736758478358383 usec, L: 0
0113 08:54:45.263707 (+ 775us) write_op.cc:270] Start()
0113 08:54:45.263731 (+ 24us) write_op.cc:276] Timestamp: P: 1736758478390691 usec, L: 0
0113 08:54:45.263905 (+ 174us) write_op.cc:270] Start()
0113 08:54:45.263979 (+ 74us) write_op.cc:276] Timestamp: P: 1736758478439786 usec, L: 0
0113 08:54:45.267457 (+ 3478us) write_op.cc:270] Start()
0113 08:54:45.267486 (+ 29us) write_op.cc:276] Timestamp: P: 1736758478442359 usec, L: 0
0113 08:54:45.268007 (+ 521us) write_op.cc:270] Start()
0113 08:54:45.268033 (+ 26us) write_op.cc:276] Timestamp: P: 1736758478444077 usec, L: 0
0113 08:54:45.268902 (+ 869us) write_op.cc:270] Start()
0113 08:54:45.268926 (+ 24us) write_op.cc:276] Timestamp: P: 1736758478513881 usec, L: 0
0113 08:54:45.269740 (+ 814us) write_op.cc:270] Start()
0113 08:54:45.269768 (+ 28us) write_op.cc:276] Timestamp: P: 1736758478546249 usec, L: 0
0113 08:54:45.273054 (+ 3286us) write_op.cc:270] Start()
0113 08:54:45.273078 (+ 24us) write_op.cc:276] Timestamp: P: 1736758478557437 usec, L: 0
0113 08:54:45.273514 (+ 436us) write_op.cc:270] Start()
0113 08:54:45.273533 (+ 19us) write_op.cc:276] Timestamp: P: 1736758478558108 usec, L: 0
0113 08:54:45.273981 (+ 448us) write_op.cc:270] Start()
0113 08:54:45.274004 (+ 23us) write_op.cc:276] Timestamp: P: 1736758478604679 usec, L: 0
0113 08:54:45.274374 (+ 370us) write_op.cc:270] Start()
0113 08:54:45.274394 (+ 20us) write_op.cc:276] Timestamp: P: 1736758478610687 usec, L: 0
0113 08:54:45.274824 (+ 430us) write_op.cc:270] Start()
0113 08:54:45.274845 (+ 21us) write_op.cc:276] Timestamp: P: 1736758478637311 usec, L: 0
0113 08:54:45.282335 (+ 7490us) write_op.cc:270] Start()
0113 08:54:45.282360 (+ 25us) write_op.cc:276] Timestamp: P: 1736758478640813 usec, L: 0
0113 08:54:45.282545 (+ 185us) write_op.cc:270] Start()
0113 08:54:45.282572 (+ 27us) write_op.cc:276] Timestamp: P: 1736758478680634 usec, L: 0
0113 08:54:45.282746 (+ 174us) write_op.cc:270] Start()
0113 08:54:45.282771 (+ 25us) write_op.cc:276] Timestamp: P: 1736758478682944 usec, L: 0
0113 08:54:45.283041 (+ 270us) write_op.cc:270] Start()
0113 08:54:45.283064 (+ 23us) write_op.cc:276] Timestamp: P: 1736758478717760 usec, L: 0
0113 08:54:45.283254 (+ 190us) write_op.cc:270] Start()
0113 08:54:45.283275 (+ 21us) write_op.cc:276] Timestamp: P: 1736758478722955 usec, L: 0
0113 08:54:45.283471 (+ 196us) write_op.cc:270] Start()
0113 08:54:45.283498 (+ 27us) write_op.cc:276] Timestamp: P: 1736758478737202 usec, L: 0
0113 08:54:45.283673 (+ 175us) write_op.cc:270] Start()
0113 08:54:45.283695 (+ 22us) write_op.cc:276] Timestamp: P: 1736758478749686 usec, L: 0
0113 08:54:45.283882 (+ 187us) write_op.cc:270] Start()
0113 08:54:45.283918 (+ 36us) write_op.cc:276] Timestamp: P: 1736758478779961 usec, L: 0
0113 08:54:45.284100 (+ 182us) write_op.cc:270] Start()
0113 08:54:45.284125 (+ 25us) write_op.cc:276] Timestamp: P: 1736758478783499 usec, L: 0
0113 08:54:45.284312 (+ 187us) write_op.cc:270] Start()
0113 08:54:45.284335 (+ 23us) write_op.cc:276] Timestamp: P: 1736758478812938 usec, L: 0
0113 08:54:45.284517 (+ 182us) write_op.cc:270] Start()
0113 08:54:45.284544 (+ 27us) write_op.cc:276] Timestamp: P: 1736758478858025 usec, L: 0
0113 08:54:45.284723 (+ 179us) write_op.cc:270] Start()
0113 08:54:45.284743 (+ 20us) write_op.cc:276] Timestamp: P: 1736758478858793 usec, L: 0
0113 08:54:45.284918 (+ 175us) write_op.cc:270] Start()
0113 08:54:45.284939 (+ 21us) write_op.cc:276] Timestamp: P: 1736758478860291 usec, L: 0
0113 08:54:45.285118 (+ 179us) write_op.cc:270] Start()
0113 08:54:45.285138 (+ 20us) write_op.cc:276] Timestamp: P: 1736758478898667 usec, L: 0
0113 08:54:45.285331 (+ 193us) write_op.cc:270] Start()
0113 08:54:45.285353 (+ 22us) write_op.cc:276] Timestamp: P: 1736758478915336 usec, L: 0
0113 08:54:45.285528 (+ 175us) write_op.cc:270] Start()
0113 08:54:45.285549 (+ 21us) write_op.cc:276] Timestamp: P: 1736758478934184 usec, L: 0
0113 08:54:45.285731 (+ 182us) write_op.cc:270] Start()
0113 08:54:45.285757 (+ 26us) write_op.cc:276] Timestamp: P: 1736758478940316 usec, L: 0
0113 08:54:45.285950 (+ 193us) write_op.cc:270] Start()
0113 08:54:45.285972 (+ 22us) write_op.cc:276] Timestamp: P: 1736758479011742 usec, L: 0
0113 08:54:45.286132 (+ 160us) write_op.cc:270] Start()
0113 08:54:45.286151 (+ 19us) write_op.cc:276] Timestamp: P: 1736758479025438 usec, L: 0
0113 08:54:45.286314 (+ 163us) write_op.cc:270] Start()
0113 08:54:45.286333 (+ 19us) write_op.cc:276] Timestamp: P: 1736758479027896 usec, L: 0
0113 08:54:45.286508 (+ 175us) write_op.cc:270] Start()
0113 08:54:45.286530 (+ 22us) write_op.cc:276] Timestamp: P: 1736758479037809 usec, L: 0
0113 08:54:45.286705 (+ 175us) write_op.cc:270] Start()
0113 08:54:45.286726 (+ 21us) write_op.cc:276] Timestamp: P: 1736758479080527 usec, L: 0
0113 08:54:45.286935 (+ 209us) write_op.cc:270] Start()
0113 08:54:45.286956 (+ 21us) write_op.cc:276] Timestamp: P: 1736758479081438 usec, L: 0
0113 08:54:45.287168 (+ 212us) write_op.cc:270] Start()
0113 08:54:45.287189 (+ 21us) write_op.cc:276] Timestamp: P: 1736758479084841 usec, L: 0
0113 08:54:45.287370 (+ 181us) write_op.cc:270] Start()
0113 08:54:45.287391 (+ 21us) write_op.cc:276] Timestamp: P: 1736758479116032 usec, L: 0
0113 08:54:45.287564 (+ 173us) write_op.cc:270] Start()
0113 08:54:45.287585 (+ 21us) write_op.cc:276] Timestamp: P: 1736758479166301 usec, L: 0
0113 08:54:45.295610 (+ 8025us) write_op.cc:270] Start()
0113 08:54:45.295652 (+ 42us) write_op.cc:276] Timestamp: P: 1736758479193092 usec, L: 0
0113 08:54:45.295976 (+ 324us) write_op.cc:270] Start()
0113 08:54:45.296001 (+ 25us) write_op.cc:276] Timestamp: P: 1736758479208868 usec, L: 0
0113 08:54:45.296188 (+ 187us) write_op.cc:270] Start()
0113 08:54:45.296212 (+ 24us) write_op.cc:276] Timestamp: P: 1736758479226761 usec, L: 0
0113 08:54:45.296499 (+ 287us) write_op.cc:270] Start()
0113 08:54:45.296522 (+ 23us) write_op.cc:276] Timestamp: P: 1736758479255593 usec, L: 0
0113 08:54:45.296732 (+ 210us) write_op.cc:270] Start()
0113 08:54:45.296757 (+ 25us) write_op.cc:276] Timestamp: P: 1736758479256564 usec, L: 0
0113 08:54:45.296961 (+ 204us) write_op.cc:270] Start()
0113 08:54:45.296988 (+ 27us) write_op.cc:276] Timestamp: P: 1736758479274049 usec, L: 0
0113 08:54:45.297198 (+ 210us) write_op.cc:270] Start()
0113 08:54:45.297221 (+ 23us) write_op.cc:276] Timestamp: P: 1736758479293256 usec, L: 0
0113 08:54:45.297415 (+ 194us) write_op.cc:270] Start()
0113 08:54:45.297436 (+ 21us) write_op.cc:276] Timestamp: P: 1736758479333392 usec, L: 0
0113 08:54:45.297622 (+ 186us) write_op.cc:270] Start()
0113 08:54:45.297644 (+ 22us) write_op.cc:276] Timestamp: P: 1736758479349898 usec, L: 0
0113 08:54:45.297838 (+ 194us) write_op.cc:270] Start()
0113 08:54:45.297860 (+ 22us) write_op.cc:276] Timestamp: P: 1736758479350582 usec, L: 0
0113 08:54:45.298081 (+ 221us) write_op.cc:270] Start()
0113 08:54:45.298102 (+ 21us) write_op.cc:276] Timestamp: P: 1736758479358035 usec, L: 0
0113 08:54:45.298368 (+ 266us) write_op.cc:270] Start()
0113 08:54:45.298386 (+ 18us) write_op.cc:276] Timestamp: P: 1736758479386888 usec, L: 0
0113 08:54:45.298568 (+ 182us) write_op.cc:270] Start()
0113 08:54:45.298587 (+ 19us) write_op.cc:276] Timestamp: P: 1736758479476794 usec, L: 0
0113 08:54:45.298771 (+ 184us) write_op.cc:270] Start()
0113 08:54:45.298787 (+ 16us) write_op.cc:276] Timestamp: P: 1736758479480153 usec, L: 0
0113 08:54:45.299004 (+ 217us) write_op.cc:270] Start()
0113 08:54:45.299020 (+ 16us) write_op.cc:276] Timestamp: P: 1736758479487462 usec, L: 0
0113 08:54:45.299488 (+ 468us) write_op.cc:270] Start()
0113 08:54:45.299511 (+ 23us) write_op.cc:276] Timestamp: P: 1736758479488060 usec, L: 0
0113 08:54:45.313335 (+ 13824us) write_op.cc:270] Start()
0113 08:54:45.313369 (+ 34us) write_op.cc:276] Timestamp: P: 1736758479536849 usec, L: 0
0113 08:54:45.314082 (+ 713us) write_op.cc:270] Start()
0113 08:54:45.314114 (+ 32us) write_op.cc:276] Timestamp: P: 1736758479579209 usec, L: 0
0113 08:54:45.315056 (+ 942us) write_op.cc:270] Start()
0113 08:54:45.315083 (+ 27us) write_op.cc:276] Timestamp: P: 1736758479588409 usec, L: 0
0113 08:54:45.385742 (+ 70659us) write_op.cc:270] Start()
0113 08:54:45.385786 (+ 44us) write_op.cc:276] Timestamp: P: 1736758479589062 usec, L: 0
0113 08:54:45.398713 (+ 12927us) write_op.cc:270] Start()
0113 08:54:45.398752 (+ 39us) write_op.cc:276] Timestamp: P: 1736758479590836 usec, L: 0
0113 08:54:45.399083 (+ 331us) write_op.cc:270] Start()
0113 08:54:45.399107 (+ 24us) write_op.cc:276] Timestamp: P: 1736758479627452 usec, L: 0
0113 08:54:45.399747 (+ 640us) write_op.cc:270] Start()
0113 08:54:45.399772 (+ 25us) write_op.cc:276] Timestamp: P: 1736758479637305 usec, L: 0
0113 08:54:45.399985 (+ 213us) write_op.cc:270] Start()
0113 08:54:45.400010 (+ 25us) write_op.cc:276] Timestamp: P: 173675
I20250113 08:54:48.108534 8787 raft_consensus.cc:491] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:48.115806 8787 raft_consensus.cc:513] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:48.172533 8787 leader_election.cc:290] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:48.174571 7967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 1530 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:48.204053 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2a45627bfe4db0b9cdccfab81e2805" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 1530 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:48.207069 8581 leader_election.cc:304] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:48.207732 8787 raft_consensus.cc:2743] T fe2a45627bfe4db0b9cdccfab81e2805 P aea19a9cd21144508fdcfd7b3570a0e1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250113 08:54:48.287770 8659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.2:42655 (request call id 45) took 4405 ms. Trace:
I20250113 08:54:48.288169 8774 raft_consensus.cc:491] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1abc86bb580b4d14876623243c9d6594)
I20250113 08:54:48.289096 8774 raft_consensus.cc:513] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:48.289943 8774 leader_election.cc:290] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:48.290920 8615 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37056 (request call id 50) took 3516 ms. Trace:
I20250113 08:54:48.291028 8615 rpcz_store.cc:276] 0113 08:54:44.774594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:44.774675 (+ 81us) service_pool.cc:225] Handling call
0113 08:54:48.290225 (+3515550us) spinlock_profiling.cc:243] Waited 3.51 s on lock 0x61600001ab54. stack: 00007f4647092fd6 00007f4647092d90 00007f464607248d 000000000096d544 000000000096d50d 000000000096ceda 00007f4652905fd0 00007f4656642b3f 00007f4650f9d8e6 00007f4650f9d615 00007f464ddfa30e 00007f464ddf91ee 00007f464ddff425 00007f464ddffd4f 00007f464ddffb78 0000000000a71ecc
0113 08:54:48.290898 (+ 673us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8080761600}
I20250113 08:54:48.294293 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35e4ecbe2cc424089eaa5e0249df483" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1354 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:48.334918 8586 tablet_bootstrap.cc:492] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=201 overwritten=0 applied=200 ignored=0} inserts{seen=1616 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 08:54:48.335495 8586 tablet_bootstrap.cc:492] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1: Bootstrap complete.
I20250113 08:54:48.336802 8586 ts_tablet_manager.cc:1397] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1: Time spent bootstrapping tablet: real 4.323s user 0.373s sys 0.049s
I20250113 08:54:48.337592 8586 raft_consensus.cc:357] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:48.338145 8586 raft_consensus.cc:738] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea19a9cd21144508fdcfd7b3570a0e1, State: Initialized, Role: FOLLOWER
I20250113 08:54:48.338397 8586 consensus_queue.cc:260] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 200, Last appended: 1.201, Last appended by leader: 201, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:48.360105 7967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35e4ecbe2cc424089eaa5e0249df483" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1354 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:48.388345 8586 ts_tablet_manager.cc:1428] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1: Time spent starting tablet: real 0.051s user 0.004s sys 0.000s
I20250113 08:54:48.391364 8583 leader_election.cc:304] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:48.391927 8774 raft_consensus.cc:2743] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:48.612936 8774 raft_consensus.cc:491] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:48.613162 8774 raft_consensus.cc:513] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:48.614014 8774 leader_election.cc:290] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:48.625320 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 201 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:48.300143 8659 rpcz_store.cc:276] 0113 08:54:43.882717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:43.883339 (+ 622us) service_pool.cc:225] Handling call
0113 08:54:44.389755 (+506416us) raft_consensus.cc:1466] Updating replica for 1153 ops
0113 08:54:45.060589 (+670834us) raft_consensus.cc:1529] Early marking committed up to index 201
0113 08:54:45.060718 (+ 129us) raft_consensus.cc:1534] Triggering prepare for 1153 ops
0113 08:54:45.060848 (+ 130us) write_op.cc:270] Start()
0113 08:54:45.060876 (+ 28us) write_op.cc:276] Timestamp: P: 1736758458199851 usec, L: 0
0113 08:54:45.061097 (+ 221us) write_op.cc:270] Start()
0113 08:54:45.061125 (+ 28us) write_op.cc:276] Timestamp: P: 1736758458203128 usec, L: 0
0113 08:54:45.065792 (+ 4667us) write_op.cc:270] Start()
0113 08:54:45.065824 (+ 32us) write_op.cc:276] Timestamp: P: 1736758458205296 usec, L: 0
0113 08:54:45.066038 (+ 214us) write_op.cc:270] Start()
0113 08:54:45.066059 (+ 21us) write_op.cc:276] Timestamp: P: 1736758458467409 usec, L: 0
0113 08:54:45.066306 (+ 247us) write_op.cc:270] Start()
0113 08:54:45.066331 (+ 25us) write_op.cc:276] Timestamp: P: 1736758458808440 usec, L: 0
0113 08:54:45.066528 (+ 197us) write_op.cc:270] Start()
0113 08:54:45.066558 (+ 30us) write_op.cc:276] Timestamp: P: 1736758458829381 usec, L: 0
0113 08:54:45.066753 (+ 195us) write_op.cc:270] Start()
0113 08:54:45.066843 (+ 90us) write_op.cc:276] Timestamp: P: 1736758458846058 usec, L: 0
0113 08:54:45.067054 (+ 211us) write_op.cc:270] Start()
0113 08:54:45.067077 (+ 23us) write_op.cc:276] Timestamp: P: 1736758458855504 usec, L: 0
0113 08:54:45.067300 (+ 223us) write_op.cc:270] Start()
0113 08:54:45.067324 (+ 24us) write_op.cc:276] Timestamp: P: 1736758458878113 usec, L: 0
0113 08:54:45.067528 (+ 204us) write_op.cc:270] Start()
0113 08:54:45.067552 (+ 24us) write_op.cc:276] Timestamp: P: 1736758458902869 usec, L: 0
0113 08:54:45.067750 (+ 198us) write_op.cc:270] Start()
0113 08:54:45.067773 (+ 23us) write_op.cc:276] Timestamp: P: 1736758458933367 usec, L: 0
0113 08:54:45.067969 (+ 196us) write_op.cc:270] Start()
0113 08:54:45.067992 (+ 23us) write_op.cc:276] Timestamp: P: 1736758458934010 usec, L: 0
0113 08:54:45.068344 (+ 352us) write_op.cc:270] Start()
0113 08:54:45.068368 (+ 24us) write_op.cc:276] Timestamp: P: 1736758458980773 usec, L: 0
0113 08:54:45.068566 (+ 198us) write_op.cc:270] Start()
0113 08:54:45.068588 (+ 22us) write_op.cc:276] Timestamp: P: 1736758458982364 usec, L: 0
0113 08:54:45.068776 (+ 188us) write_op.cc:270] Start()
0113 08:54:45.068798 (+ 22us) write_op.cc:276] Timestamp: P: 1736758458991907 usec, L: 0
0113 08:54:45.069015 (+ 217us) write_op.cc:270] Start()
0113 08:54:45.069049 (+ 34us) write_op.cc:276] Timestamp: P: 1736758458992884 usec, L: 0
0113 08:54:45.069288 (+ 239us) write_op.cc:270] Start()
0113 08:54:45.069312 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459039410 usec, L: 0
0113 08:54:45.069508 (+ 196us) write_op.cc:270] Start()
0113 08:54:45.069530 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459044294 usec, L: 0
0113 08:54:45.069727 (+ 197us) write_op.cc:270] Start()
0113 08:54:45.069749 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459048108 usec, L: 0
0113 08:54:45.069945 (+ 196us) write_op.cc:270] Start()
0113 08:54:45.069968 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459049711 usec, L: 0
0113 08:54:45.070169 (+ 201us) write_op.cc:270] Start()
0113 08:54:45.070192 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459087411 usec, L: 0
0113 08:54:45.070390 (+ 198us) write_op.cc:270] Start()
0113 08:54:45.070413 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459106967 usec, L: 0
0113 08:54:45.070602 (+ 189us) write_op.cc:270] Start()
0113 08:54:45.070624 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459109699 usec, L: 0
0113 08:54:45.070829 (+ 205us) write_op.cc:270] Start()
0113 08:54:45.070850 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459110426 usec, L: 0
0113 08:54:45.071058 (+ 208us) write_op.cc:270] Start()
0113 08:54:45.071080 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459129931 usec, L: 0
0113 08:54:45.071282 (+ 202us) write_op.cc:270] Start()
0113 08:54:45.071304 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459170009 usec, L: 0
0113 08:54:45.071486 (+ 182us) write_op.cc:270] Start()
0113 08:54:45.071507 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459179989 usec, L: 0
0113 08:54:45.071712 (+ 205us) write_op.cc:270] Start()
0113 08:54:45.071734 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459188207 usec, L: 0
0113 08:54:45.071925 (+ 191us) write_op.cc:270] Start()
0113 08:54:45.071947 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459190746 usec, L: 0
0113 08:54:45.072152 (+ 205us) write_op.cc:270] Start()
0113 08:54:45.072177 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459237289 usec, L: 0
0113 08:54:45.072378 (+ 201us) write_op.cc:270] Start()
0113 08:54:45.072401 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459238913 usec, L: 0
0113 08:54:45.072586 (+ 185us) write_op.cc:270] Start()
0113 08:54:45.072608 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459251759 usec, L: 0
0113 08:54:45.072835 (+ 227us) write_op.cc:270] Start()
0113 08:54:45.072859 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459254650 usec, L: 0
0113 08:54:45.073089 (+ 230us) write_op.cc:270] Start()
0113 08:54:45.073111 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459275748 usec, L: 0
0113 08:54:45.074014 (+ 903us) write_op.cc:270] Start()
0113 08:54:45.074039 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459284652 usec, L: 0
0113 08:54:45.074234 (+ 195us) write_op.cc:270] Start()
0113 08:54:45.074268 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459304532 usec, L: 0
0113 08:54:45.074467 (+ 199us) write_op.cc:270] Start()
0113 08:54:45.074487 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459310885 usec, L: 0
0113 08:54:45.074666 (+ 179us) write_op.cc:270] Start()
0113 08:54:45.074686 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459334393 usec, L: 0
0113 08:54:45.074889 (+ 203us) write_op.cc:270] Start()
0113 08:54:45.074910 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459355653 usec, L: 0
0113 08:54:45.075099 (+ 189us) write_op.cc:270] Start()
0113 08:54:45.075120 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459357727 usec, L: 0
0113 08:54:45.075316 (+ 196us) write_op.cc:270] Start()
0113 08:54:45.075337 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459417602 usec, L: 0
0113 08:54:45.075505 (+ 168us) write_op.cc:270] Start()
0113 08:54:45.075525 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459419973 usec, L: 0
0113 08:54:45.075704 (+ 179us) write_op.cc:270] Start()
0113 08:54:45.075723 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459422980 usec, L: 0
0113 08:54:45.075898 (+ 175us) write_op.cc:270] Start()
0113 08:54:45.075919 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459427897 usec, L: 0
0113 08:54:45.076108 (+ 189us) write_op.cc:270] Start()
0113 08:54:45.076129 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459461607 usec, L: 0
0113 08:54:45.076321 (+ 192us) write_op.cc:270] Start()
0113 08:54:45.076343 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459482100 usec, L: 0
0113 08:54:45.076662 (+ 319us) write_op.cc:270] Start()
0113 08:54:45.076685 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459486713 usec, L: 0
0113 08:54:45.076878 (+ 193us) write_op.cc:270] Start()
0113 08:54:45.076900 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459505831 usec, L: 0
0113 08:54:45.077083 (+ 183us) write_op.cc:270] Start()
0113 08:54:45.077106 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459508953 usec, L: 0
0113 08:54:45.077315 (+ 209us) write_op.cc:270] Start()
0113 08:54:45.077338 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459547378 usec, L: 0
0113 08:54:45.077524 (+ 186us) write_op.cc:270] Start()
0113 08:54:45.077544 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459548158 usec, L: 0
0113 08:54:45.077733 (+ 189us) write_op.cc:270] Start()
0113 08:54:45.077759 (+ 26us) write_op.cc:276] Timestamp: P: 1736758459570717 usec, L: 0
0113 08:54:45.077946 (+ 187us) write_op.cc:270] Start()
0113 08:54:45.077967 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459583990 usec, L: 0
0113 08:54:45.078153 (+ 186us) write_op.cc:270] Start()
0113 08:54:45.078175 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459585325 usec, L: 0
0113 08:54:45.078353 (+ 178us) write_op.cc:270] Start()
0113 08:54:45.078373 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459605050 usec, L: 0
0113 08:54:45.078544 (+ 171us) write_op.cc:270] Start()
0113 08:54:45.078565 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459636072 usec, L: 0
0113 08:54:45.078736 (+ 171us) write_op.cc:270] Start()
0113 08:54:45.078761 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459644178 usec, L: 0
0113 08:54:45.078963 (+ 202us) write_op.cc:270] Start()
0113 08:54:45.078984 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459650990 usec, L: 0
0113 08:54:45.079173 (+ 189us) write_op.cc:270] Start()
0113 08:54:45.079192 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459665467 usec, L: 0
0113 08:54:45.079378 (+ 186us) write_op.cc:270] Start()
0113 08:54:45.079421 (+ 43us) write_op.cc:276] Timestamp: P: 1736758459697798 usec, L: 0
0113 08:54:45.079597 (+ 176us) write_op.cc:270] Start()
0113 08:54:45.079617 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459712660 usec, L: 0
0113 08:54:45.079797 (+ 180us) write_op.cc:270] Start()
0113 08:54:45.079818 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459714950 usec, L: 0
0113 08:54:45.080010 (+ 192us) write_op.cc:270] Start()
0113 08:54:45.080032 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459721639 usec, L: 0
0113 08:54:45.080227 (+ 195us) write_op.cc:270] Start()
0113 08:54:45.080251 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459740273 usec, L: 0
0113 08:54:45.080477 (+ 226us) write_op.cc:270] Start()
0113 08:54:45.080501 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459790287 usec, L: 0
0113 08:54:45.080692 (+ 191us) write_op.cc:270] Start()
0113 08:54:45.080713 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459791556 usec, L: 0
0113 08:54:45.080879 (+ 166us) write_op.cc:270] Start()
0113 08:54:45.080899 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459799634 usec, L: 0
0113 08:54:45.082672 (+ 1773us) write_op.cc:270] Start()
0113 08:54:45.082700 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459802405 usec, L: 0
0113 08:54:45.084704 (+ 2004us) write_op.cc:270] Start()
0113 08:54:45.084738 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459844275 usec, L: 0
0113 08:54:45.084951 (+ 213us) write_op.cc:270] Start()
0113 08:54:45.084975 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459848200 usec, L: 0
0113 08:54:45.085170 (+ 195us) write_op.cc:270] Start()
0113 08:54:45.085190 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459873849 usec, L: 0
0113 08:54:45.103427 (+ 18237us) write_op.cc:270] Start()
0113 08:54:45.103456 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459875014 usec, L: 0
0113 08:54:45.113179 (+ 9723us) write_op.cc:270] Start()
0113 08:54:45.113223 (+ 44us) write_op.cc:276] Timestamp: P: 1736758459903820 usec, L: 0
0113 08:54:45.113471 (+ 248us) write_op.cc:270] Start()
0113 08:54:45.113500 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459916744 usec, L: 0
0113 08:54:45.113735 (+ 235us) write_op.cc:270] Start()
0113 08:54:45.113758 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459943131 usec, L: 0
0113 08:54:45.114004 (+ 246us) write_op.cc:270] Start()
0113 08:54:45.114028 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459952096 usec, L: 0
0113 08:54:45.114210 (+ 182us) write_op.cc:270] Start()
0113 08:54:45.114232 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459981368 usec, L: 0
0113 08:54:45.114393 (+ 161us) write_op.cc:270] Start()
0113 08:54:45.114437 (+ 44us) write_op.cc:276] Timestamp: P: 1736758459983121 usec, L: 0
0113 08:54:45.114906 (+ 469us) write_op.cc:270] Start()
0113 08:54:45.114934 (+ 28us) write_op.cc:276] Timestamp: P: 1736758459988927 usec, L: 0
0113 08:54:45.115083 (+ 149us) write_op.cc:270] Start()
0113 08:54:45.115105 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460020137 usec, L: 0
0113 08:54:45.115267 (+ 162us) write_op.cc:270] Start()
0113 08:54:45.115290 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460043371 usec, L: 0
0113 08:54:45.115449 (+ 159us) write_op.cc:270] Start()
0113 08:54:45.115471 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460044873 usec, L: 0
0113 08:54:45.115620 (+ 149us) write_op.cc:270] Start()
0113 08:54:45.115643 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460062984 usec, L: 0
0113 08:54:45.115792 (+ 149us) write_op.cc:270] Start()
0113 08:54:45.115813 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460065857 usec, L: 0
0113 08:54:45.116000 (+ 187us) write_op.cc:270] Start()
0113 08:54:45.116022 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460100047 usec, L: 0
0113 08:54:45.116163 (+ 141us) write_op.cc:270] Start()
0113 08:54:45.116182 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460101511 usec, L: 0
0113 08:54:45.116308 (+ 126us) write_op.cc:270] Start()
0113 08:54:45.116326 (+ 18us) write_op.cc:276] Timestamp: P: 1736758460120624 usec, L: 0
0113 08:54:45.116463 (+ 137us) write_op.cc:270] Start()
0113 08:54:45.116485 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460133317 usec, L: 0
0113 08:54:45.116766 (+ 281us) write_op.cc:270] Start()
0113 08:54:45.116793 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460156073 usec, L: 0
0113 08:54:45.117007 (+ 214us) write_op.cc:270] Start()
0113 08:54:45.117028 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460163838 usec, L: 0
0113 08:54:45.117177 (+ 149us) write_op.cc:270] Start()
0113 08:54:45.117197 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460186369 usec, L: 0
0113 08:54:45.117348 (+ 151us) write_op.cc:270] Start()
0113 08:54:45.117370 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460190756 usec, L: 0
0113 08:54:45.117520 (+ 150us) write_op.cc:270] Start()
0113 08:54:45.117542 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460221249 usec, L: 0
0113 08:54:45.117700 (+ 158us) write_op.cc:270] Start()
0113 08:54:45.117722 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460233898 usec, L: 0
0113 08:54:45.118041 (+ 319us) write_op.cc:270] Start()
0113 08:54:45.118067 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460248836 usec, L: 0
0113 08:54:45.118255 (+ 188us) write_op.cc:270] Start()
0113 08:54:45.118279 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460249516 usec, L: 0
0113 08:54:45.118438 (+ 159us) write_op.cc:270] Start()
0113 08:54:45.118461 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460283163 usec, L: 0
0113 08:54:45.118614 (+ 153us) write_op.cc:270] Start()
0113 08:54:45.118635 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460284315 usec, L: 0
0113 08:54:45.118842 (+ 207us) write_op.cc:270] Start()
0113 08:54:45.118866 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460307047 usec, L: 0
0113 08:54:45.119049 (+ 183us) write_op.cc:270] Start()
0113 08:54:45.119074 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460337622 usec, L: 0
0113 08:54:45.119231 (+ 157us) write_op.cc:270] Start()
0113 08:54:45.119254 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460340823 usec, L: 0
0113 08:54:45.119407 (+ 153us) write_op.cc:270] Start()
0113 08:54:45.119429 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460346893 usec, L: 0
0113 08:54:45.119588 (+ 159us) write_op.cc:270] Start()
0113 08:54:45.119610 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460348464 usec, L: 0
0113 08:54:45.119789 (+ 179us) write_op.cc:270] Start()
0113 08:54:45.119812 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460393515 usec, L: 0
0113 08:54:45.119981 (+ 169us) write_op.cc:270] Start()
0113 08:54:45.120003 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460395982 usec, L: 0
0113 08:54:45.120266 (+ 263us) write_op.cc:270] Start()
0113 08:54:45.120291 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460412986 usec, L: 0
0113 08:54:45.120485 (+ 194us) write_op.cc:270] Start()
0113 08:54:45.120509 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460419223 usec, L: 0
0113 08:54:45.120701 (+ 192us) write_op.cc:270] Start()
0113 08:54:45.120725 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460457890 usec, L: 0
0113 08:54:45.120930 (+ 205us) write_op.cc:270] Start()
0113 08:54:45.120954 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460476034 usec, L: 0
0113 08:54:45.121149 (+ 195us) write_op.cc:270] Start()
0113 08:54:45.121172 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460482637 usec, L: 0
0113 08:54:45.121354 (+ 182us) write_op.cc:270] Start()
0113 08:54:45.121374 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460503507 usec, L: 0
0113 08:54:45.121543 (+ 169us) write_op.cc:270] Start()
0113 08:54:45.121564 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460531377 usec, L: 0
0113 08:54:45.121806 (+ 242us) write_op.cc:270] Start()
0113 08:54:45.121845 (+ 39us) write_op.cc:276] Timestamp: P: 1736758460555631 usec, L: 0
0113 08:54:45.122084 (+ 239us) write_op.cc:270] Start()
0113 08:54:45.122111 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460562470 usec, L: 0
0113 08:54:45.122295 (+ 184us) write_op.cc:270] Start()
0113 08:54:45.122317 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460578672 usec, L: 0
0113 08:54:45.122526 (+ 209us) write_op.cc:270] Start()
0113 08:54:45.122550 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460605453 usec, L: 0
0113 08:54:45.122760 (+ 210us) write_op.cc:270] Start()
0113 08:54:45.122784 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460608095 usec, L: 0
0113 08:54:45.123023 (+ 239us) write_op.cc:270] Start()
0113 08:54:45.123047 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460622740 usec, L: 0
0113 08:54:45.123234 (+ 187us) write_op.cc:270] Start()
0113 08:54:45.123256 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460624656 usec, L: 0
0113 08:54:45.123479 (+ 223us) write_op.cc:270] Start()
0113 08:54:45.123502 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460651430 usec, L: 0
0113 08:54:45.123697 (+ 195us) write_op.cc:270] Start()
0113 08:54:45.123720 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460678509 usec, L: 0
0113 08:54:45.123893 (+ 173us) write_op.cc:270] Start()
0113 08:54:45.123929 (+ 36us) write_op.cc:276] Timestamp: P: 1736758460700892 usec, L: 0
0113 08:54:45.127021 (+ 3092us) write_op.cc:270] Start()
0113 08:54:45.127050 (+ 29us) write_op.cc:276] Timestamp: P: 1736758460704421 usec, L: 0
0113 08:54:45.127263 (+ 213us) write_op.cc:270] Start()
0113 08:54:45.127285 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460705233 usec, L: 0
0113 08:54:45.127492 (+ 207us) write_op.cc:270] Start()
0113 08:54:45.127518 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460721473 usec, L: 0
0113 08:54:45.127716 (+ 198us) write_op.cc:270] Start()
0113 08:54:45.127742 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460768223 usec, L: 0
0113 08:54:45.127954 (+ 212us) write_op.cc:270] Start()
0113 08:54:45.127978 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460769111 usec, L: 0
0113 08:54:45.128140 (+ 162us) write_op.cc:270] Start()
0113 08:54:45.128163 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460771844 usec, L: 0
0113 08:54:45.128371 (+ 208us) write_op.cc:270] Start()
0113 08:54:45.128395 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460810542 usec, L: 0
0113 08:54:45.128558 (+ 163us) write_op.cc:270] Start()
0113 08:54:45.128582 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460820615 usec, L: 0
0113 08:54:45.128735 (+ 153us) write_op.cc:270] Start()
0113 08:54:45.128774 (+ 39us) write_op.cc:276] Timestamp: P: 1736758460823385 usec, L: 0
0113 08:54:45.128939 (+ 165us) write_op.cc:270] Start()
0113 08:54:45.128963 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460827770 usec, L: 0
0113 08:54:45.129117 (+ 154us) write_op.cc:270] Start()
0113 08:54:45.129140 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460870411 usec, L: 0
0113 08:54:45.129310 (+ 170us) write_op.cc:270] Start()
0113 08:54:45.129335 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460884925 usec, L: 0
0113 08:54:45.129495 (+ 160us) write_op.cc:270] Start()
0113 08:54:45.129517 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460886287 usec, L: 0
0113 08:54:45.129671 (+ 154us) write_op.cc:270] Start()
0113 08:54:45.129695 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460901048 usec, L: 0
0113 08:54:45.129851 (+ 156us) write_op.cc:270] Start()
0113 08:54:45.129873 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460928012 usec, L: 0
0113 08:54:45.130061 (+ 188us) write_op.cc:270] Start()
0113 08:54:45.130083 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460938949 usec, L: 0
0113 08:54:45.143336 (+ 13253us) write_op.cc:270] Start()
0113 08:54:45.143389 (+ 53us) write_op.cc:276] Timestamp: P: 1736758460948521 usec, L: 0
0113 08:54:45.160018 (+ 16629us) write_op.cc:270] Start()
0113 08:54:45.160045 (+ 27us) write_op.cc:276] Timestamp: P: 1736758460962162 usec, L: 0
0113 08:54:45.167175 (+ 7130us) write_op.cc:270] Start()
0113 08:54:45.167209 (+ 34us) write_op.cc:276] Timestamp: P: 1736758460995668 usec, L: 0
0113 08:54:45.167432 (+ 223us) write_op.cc:270] Start()
0113 08:54:45.167459 (+ 27us) write_op.cc:276] Timestamp: P: 1736758461010667 usec, L: 0
0113 08:54:45.167656 (+ 197us) write_op.cc:270] Start()
0113 08:54:45.167678 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461011413 usec, L: 0
0113 08:54:45.167828 (+ 150us) write_op.cc:270] Start()
0113 08:54:45.167849 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461021183 usec, L: 0
0113 08:54:45.168002 (+ 153us) write_op.cc:270] Start()
0113 08:54:45.168023 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461064642 usec, L: 0
0113 08:54:45.168155 (+ 132us) write_op.cc:270] Start()
0113 08:54:45.168175 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461073488 usec, L: 0
0113 08:54:45.168306 (+ 131us) write_op.cc:270] Start()
0113 08:54:45.168326 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461093781 usec, L: 0
0113 08:54:45.168537 (+ 211us) write_op.cc:270] Start()
0113 08:54:45.168564 (+ 27us) write_op.cc:276] Timestamp: P: 1736758461096266 usec, L: 0
0113 08:54:45.168714 (+ 150us) write_op.cc:270] Start()
0113 08:54:45.168735 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461114574 usec, L: 0
0113 08:54:45.168880 (+ 145us) write_op.cc:270] Start()
0113 08:54:45.168916 (+ 36us) write_op.cc:276] Timestamp: P: 1736758461118437 usec, L: 0
0113 08:54:45.169065 (+ 149us) write_op.cc:270] Start()
0113 08:54:45.169087 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461159745 usec, L: 0
0113 08:54:45.169229 (+ 142us) write_op.cc:270] Start()
0113 08:54:45.169248 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461167179 usec, L: 0
0113 08:54:45.169388 (+ 140us) write_op.cc:270] Start()
0113 08:54:45.169408 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461173452 usec, L: 0
0113 08:54:45.169545 (+ 137us) write_op.cc:270] Start()
0113 08:54:45.169570 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461181993 usec, L: 0
0113 08:54:45.169723 (+ 153us) write_op.cc:270] Start()
0113 08:54:45.169745 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461227635 usec, L: 0
0113 08:54:45.169915 (+ 170us) write_op.cc:270] Start()
0113 08:54:45.169938 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461232089 usec, L: 0
0113 08:54:45.170096 (+ 158us) write_op.cc:270] Start()
0113 08:54:45.170118 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461237700 usec, L: 0
0113 08:54:45.170274 (+ 156us) write_op.cc:270] Start()
0113 08:54:45.170298 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461244290 usec, L: 0
0113 08:54:45.170454 (+ 156us) write_op.cc:270] Start()
0113 08:54:45.170474 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461281178 usec, L: 0
0113 08:54:45.170742 (+ 268us) write_op.cc:270] Start()
0113 08:54:45.170765 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461282004 usec, L: 0
0113 08:54:45.171174 (+ 409us) write_op.cc:270] Start()
0113 08:54:45.171204 (+ 30us) write_op.cc:276] Timestamp: P: 1736758461297284 usec, L: 0
0113 08:54:45.171429 (+ 225us) write_op.cc:270] Start()
0113 08:54:45.171455 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461303869 usec, L: 0
0113 08:54:45.171681 (+ 226us) write_op.cc:270] Start()
0113 08:54:45.171705 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461337747 usec, L: 0
0113 08:54:45.171935 (+ 230us) write_op.cc:270] Start()
0113 08:54:45.171991 (+ 56us) write_op.cc:276] Timestamp: P: 1736758461339246 usec, L: 0
0113 08:54:45.172206 (+ 215us) write_op.cc:270] Start()
0113 08:54:45.172230 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461356727 usec, L: 0
0113 08:54:45.172449 (+ 219us) write_op.cc:270] Start()
0113 08:54:45.172524 (+ 75us) write_op.cc:276] Timestamp: P: 1736758461358599 usec, L: 0
0113 08:54:45.173075 (+ 551us) write_op.cc:270] Start()
0113 08:54:45.173099 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461392465 usec, L: 0
0113 08:54:45.173301 (+ 202us) write_op.cc:270] Start()
0113 08:54:45.173326 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461393313 usec, L: 0
0113 08:54:45.173524 (+ 198us) write_op.cc:270] Start()
0113 08:54:45.173546 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461396029 usec, L: 0
0113 08:54:45.173751 (+ 205us) write_op.cc:270] Start()
0113 08:54:45.173774 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461429041 usec, L: 0
0113 08:54:45.173995 (+ 221us) write_op.cc:270] Start()
0113 08:54:45.174017 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461464303 usec, L: 0
0113 08:54:45.174704 (+ 687us) write_op.cc:270] Start()
0113 08:54:45.174730 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461482200 usec, L: 0
0113 08:54:45.174934 (+ 204us) write_op.cc:270] Start()
0113 08:54:45.174956 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461484520 usec, L: 0
0113 08:54:45.175134 (+ 178us) write_op.cc:270] Start()
0113 08:54:45.175165 (+ 31us) write_op.cc:276] Timestamp: P: 1736758461499269 usec, L: 0
0113 08:54:45.175355 (+ 190us) write_op.cc:270] Start()
0113 08:54:45.175377 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461540816 usec, L: 0
0113 08:54:45.175552 (+ 175us) write_op.cc:270] Start()
0113 08:54:45.175569 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461541702 usec, L: 0
0113 08:54:45.175700 (+ 131us) write_op.cc:270] Start()
0113 08:54:45.175718 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461551267 usec, L: 0
0113 08:54:45.175844 (+ 126us) write_op.cc:270] Start()
0113 08:54:45.175860 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461582655 usec, L: 0
0113 08:54:45.175984 (+ 124us) write_op.cc:270] Start()
0113 08:54:45.176001 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461606322 usec, L: 0
0113 08:54:45.176159 (+ 158us) write_op.cc:270] Start()
0113 08:54:45.176176 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461633314 usec, L: 0
0113 08:54:45.176291 (+ 115us) write_op.cc:270] Start()
0113 08:54:45.176307 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461637423 usec, L: 0
0113 08:54:45.176416 (+ 109us) write_op.cc:270] Start()
0113 08:54:45.176432 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461650110 usec, L: 0
0113 08:54:45.176562 (+ 130us) write_op.cc:270] Start()
0113 08:54:45.176578 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461670236 usec, L: 0
0113 08:54:45.176697 (+ 119us) write_op.cc:270] Start()
0113 08:54:45.176713 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461702115 usec, L: 0
0113 08:54:45.176824 (+ 111us) write_op.cc:270] Start()
0113 08:54:45.176840 (+ 16us) write_op.cc:276] Timestamp: P: 1736758461708471 usec, L: 0
0113 08:54:45.176951 (+ 111us) write_op.cc:270] Start()
0113 08:54:45.176968 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461718845 usec, L: 0
0113 08:54:45.177077 (+ 109us) write_op.cc:270] Start()
0113 08:54:45.177092 (+ 15us) write_op.cc:276] Timestamp: P: 1736758461738008 usec, L: 0
0113 08:54:45.177210 (+ 118us) write_op.cc:270] Start()
0113 08:54:45.177227 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461765233 usec, L: 0
0113 08:54:45.177342 (+ 115us) write_op.cc:270] Start()
0113 08:54:45.177359 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461791843 usec, L: 0
0113 08:54:45.177475 (+ 116us) write_op.cc:270] Start()
0113 08:54:45.177492 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461796130 usec, L: 0
0113 08:54:45.177609 (+ 117us) write_op.cc:270] Start()
0113 08:54:45.177626 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461825775 usec, L: 0
0113 08:54:45.177738 (+ 112us) write_op.cc:270] Start()
0113 08:54:45.177757 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461848418 usec, L: 0
0113 08:54:45.177881 (+ 124us) write_op.cc:270] Start()
0113 08:54:45.177901 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461885712 usec, L: 0
0113 08:54:45.178033 (+ 132us) write_op.cc:270] Start()
0113 08:54:45.178055 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461903866 usec, L: 0
0113 08:54:45.178209 (+ 154us) write_op.cc:270] Start()
0113 08:54:45.178228 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461912087 usec, L: 0
0113 08:54:45.178347 (+ 119us) write_op.cc:270] Start()
0113 08:54:45.178365 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461925373 usec, L: 0
0113 08:54:45.178500 (+ 135us) write_op.cc:270] Start()
0113 08:54:45.178518 (+ 18us) write_op.cc:276] Timestamp: P: 1736
I20250113 08:54:48.678498 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 201 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:48.684597 8581 leader_election.cc:304] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:48.685057 8774 raft_consensus.cc:2743] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:48.737287 8774 raft_consensus.cc:491] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1abc86bb580b4d14876623243c9d6594)
I20250113 08:54:48.737504 8774 raft_consensus.cc:513] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:48.738260 8774 leader_election.cc:290] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:48.746965 7967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35e4ecbe2cc424089eaa5e0249df483" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1354 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:48.748690 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35e4ecbe2cc424089eaa5e0249df483" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1354 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:48.760452 8581 leader_election.cc:304] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:48.882006 8774 raft_consensus.cc:2743] T c35e4ecbe2cc424089eaa5e0249df483 P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:49.341619 8232 meta_cache.cc:1261] Time spent looking up entry by key: real 0.068s user 0.001s sys 0.000s
I20250113 08:54:49.587522 8796 raft_consensus.cc:491] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:54:49.594511 8796 raft_consensus.cc:513] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:49.604684 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1561 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:49.614991 8796 leader_election.cc:290] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:49.632014 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e64a047ae649ac9afc1b77608c067f" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1561 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:49.638739 8581 leader_election.cc:304] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:49.639390 8796 raft_consensus.cc:2743] T c1e64a047ae649ac9afc1b77608c067f P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:51.208774 8774 raft_consensus.cc:491] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1abc86bb580b4d14876623243c9d6594)
I20250113 08:54:51.208976 8774 raft_consensus.cc:513] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:51.209406 8656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.123.2:42655 (request call id 54) took 2529 ms. Trace:
I20250113 08:54:51.209882 8774 leader_election.cc:290] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:51.212289 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1532 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:51.223569 8615 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37056 (request call id 56) took 1614 ms. Trace:
I20250113 08:54:51.223709 8615 rpcz_store.cc:276] 0113 08:54:49.609266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:49.609331 (+ 65us) service_pool.cc:225] Handling call
0113 08:54:51.207803 (+1598472us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x61600001b154. stack: 00007f4647092fd6 00007f4647092d90 00007f464607248d 000000000096d544 000000000096d50d 000000000096ceda 00007f4652905fd0 00007f4656642b3f 00007f4650f9d8e6 00007f4650f9d615 00007f464ddfa30e 00007f464ddf91ee 00007f464ddff425 00007f464ddffd4f 00007f464ddffb78 0000000000a71ecc
0113 08:54:51.222960 (+ 15157us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x616000019f54. stack: 00007f4647092fd6 00007f4647092d90 00007f464607248d 000000000096d544 000000000096d50d 000000000096ceda 00007f4652905fd0 00007f4656642b3f 00007f4650f9d8e6 00007f4650f9d615 00007f464ddfa30e 00007f464ddf91ee 00007f464ddff425 00007f464ddffd4f 00007f464ddffb78 0000000000a71ecc
0113 08:54:51.223534 (+ 574us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3708382336}
I20250113 08:54:51.236424 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1532 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:51.241143 8583 leader_election.cc:304] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:51.242203 8774 raft_consensus.cc:2743] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:51.594959 8774 raft_consensus.cc:491] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1abc86bb580b4d14876623243c9d6594)
I20250113 08:54:51.595165 8774 raft_consensus.cc:513] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:51.595983 8774 leader_election.cc:290] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:51.597839 7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1532 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:51.603542 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1532 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:51.651134 8581 leader_election.cc:304] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:51.651429 8774 raft_consensus.cc:2743] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250113 08:54:51.947302 8038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.123.2:45341, user_credentials={real_user=slave}} blocked reactor thread for 44088.5us
I20250113 08:54:52.001636 8774 raft_consensus.cc:491] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1abc86bb580b4d14876623243c9d6594)
I20250113 08:54:52.001838 8774 raft_consensus.cc:513] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } }
I20250113 08:54:52.002547 8774 leader_election.cc:290] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:54:52.009820 7967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1532 } } ignore_live_leader: false dest_uuid: "1abc86bb580b4d14876623243c9d6594" is_pre_election: true
I20250113 08:54:52.026124 8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993904453c594e2db077d13f62d9b27a" candidate_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1532 } } ignore_live_leader: false dest_uuid: "81f9c074161846b1b2a552cb12d3ef30" is_pre_election: true
I20250113 08:54:52.168889 8581 leader_election.cc:304] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aea19a9cd21144508fdcfd7b3570a0e1; no voters: 1abc86bb580b4d14876623243c9d6594, 81f9c074161846b1b2a552cb12d3ef30
I20250113 08:54:52.169319 8774 raft_consensus.cc:2743] T 993904453c594e2db077d13f62d9b27a P aea19a9cd21144508fdcfd7b3570a0e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:54:51.209491 8656 rpcz_store.cc:276] 0113 08:54:48.680331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:54:48.680399 (+ 68us) service_pool.cc:225] Handling call
0113 08:54:48.793161 (+112762us) raft_consensus.cc:1466] Updating replica for 1331 ops
0113 08:54:48.897779 (+104618us) raft_consensus.cc:1529] Early marking committed up to index 201
0113 08:54:48.897853 (+ 74us) raft_consensus.cc:1534] Triggering prepare for 1331 ops
0113 08:54:48.897971 (+ 118us) write_op.cc:270] Start()
0113 08:54:48.897994 (+ 23us) write_op.cc:276] Timestamp: P: 1736758458202103 usec, L: 0
0113 08:54:48.898563 (+ 569us) write_op.cc:270] Start()
0113 08:54:48.898592 (+ 29us) write_op.cc:276] Timestamp: P: 1736758458202785 usec, L: 0
0113 08:54:48.899268 (+ 676us) write_op.cc:270] Start()
0113 08:54:48.899291 (+ 23us) write_op.cc:276] Timestamp: P: 1736758458206254 usec, L: 0
0113 08:54:48.908999 (+ 9708us) write_op.cc:270] Start()
0113 08:54:48.909042 (+ 43us) write_op.cc:276] Timestamp: P: 1736758458471430 usec, L: 0
0113 08:54:48.910665 (+ 1623us) write_op.cc:270] Start()
0113 08:54:48.910693 (+ 28us) write_op.cc:276] Timestamp: P: 1736758458813967 usec, L: 0
0113 08:54:48.911395 (+ 702us) write_op.cc:270] Start()
0113 08:54:48.911429 (+ 34us) write_op.cc:276] Timestamp: P: 1736758458847320 usec, L: 0
0113 08:54:48.912067 (+ 638us) write_op.cc:270] Start()
0113 08:54:48.912103 (+ 36us) write_op.cc:276] Timestamp: P: 1736758458848301 usec, L: 0
0113 08:54:48.912268 (+ 165us) write_op.cc:270] Start()
0113 08:54:48.912288 (+ 20us) write_op.cc:276] Timestamp: P: 1736758458855119 usec, L: 0
0113 08:54:48.912490 (+ 202us) write_op.cc:270] Start()
0113 08:54:48.912511 (+ 21us) write_op.cc:276] Timestamp: P: 1736758458878366 usec, L: 0
0113 08:54:48.912699 (+ 188us) write_op.cc:270] Start()
0113 08:54:48.912719 (+ 20us) write_op.cc:276] Timestamp: P: 1736758458905465 usec, L: 0
0113 08:54:48.912922 (+ 203us) write_op.cc:270] Start()
0113 08:54:48.912943 (+ 21us) write_op.cc:276] Timestamp: P: 1736758458933694 usec, L: 0
0113 08:54:48.913132 (+ 189us) write_op.cc:270] Start()
0113 08:54:48.913152 (+ 20us) write_op.cc:276] Timestamp: P: 1736758458934311 usec, L: 0
0113 08:54:48.913327 (+ 175us) write_op.cc:270] Start()
0113 08:54:48.913346 (+ 19us) write_op.cc:276] Timestamp: P: 1736758458974559 usec, L: 0
0113 08:54:48.913514 (+ 168us) write_op.cc:270] Start()
0113 08:54:48.913534 (+ 20us) write_op.cc:276] Timestamp: P: 1736758458980735 usec, L: 0
0113 08:54:48.913711 (+ 177us) write_op.cc:270] Start()
0113 08:54:48.913731 (+ 20us) write_op.cc:276] Timestamp: P: 1736758458991657 usec, L: 0
0113 08:54:48.913894 (+ 163us) write_op.cc:270] Start()
0113 08:54:48.913920 (+ 26us) write_op.cc:276] Timestamp: P: 1736758458992727 usec, L: 0
0113 08:54:48.914130 (+ 210us) write_op.cc:270] Start()
0113 08:54:48.914156 (+ 26us) write_op.cc:276] Timestamp: P: 1736758459035694 usec, L: 0
0113 08:54:48.914323 (+ 167us) write_op.cc:270] Start()
0113 08:54:48.914343 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459043557 usec, L: 0
0113 08:54:48.914496 (+ 153us) write_op.cc:270] Start()
0113 08:54:48.914516 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459046457 usec, L: 0
0113 08:54:48.914695 (+ 179us) write_op.cc:270] Start()
0113 08:54:48.914717 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459050859 usec, L: 0
0113 08:54:48.914911 (+ 194us) write_op.cc:270] Start()
0113 08:54:48.914933 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459089579 usec, L: 0
0113 08:54:48.915092 (+ 159us) write_op.cc:270] Start()
0113 08:54:48.915115 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459109543 usec, L: 0
0113 08:54:48.915314 (+ 199us) write_op.cc:270] Start()
0113 08:54:48.915335 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459110443 usec, L: 0
0113 08:54:48.915522 (+ 187us) write_op.cc:270] Start()
0113 08:54:48.915543 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459111235 usec, L: 0
0113 08:54:48.915742 (+ 199us) write_op.cc:270] Start()
0113 08:54:48.915763 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459130017 usec, L: 0
0113 08:54:48.915939 (+ 176us) write_op.cc:270] Start()
0113 08:54:48.915959 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459171765 usec, L: 0
0113 08:54:48.916154 (+ 195us) write_op.cc:270] Start()
0113 08:54:48.916176 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459179950 usec, L: 0
0113 08:54:48.916375 (+ 199us) write_op.cc:270] Start()
0113 08:54:48.916396 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459187391 usec, L: 0
0113 08:54:48.916592 (+ 196us) write_op.cc:270] Start()
0113 08:54:48.916612 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459191189 usec, L: 0
0113 08:54:48.916788 (+ 176us) write_op.cc:270] Start()
0113 08:54:48.916810 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459237257 usec, L: 0
0113 08:54:48.916994 (+ 184us) write_op.cc:270] Start()
0113 08:54:48.917018 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459238475 usec, L: 0
0113 08:54:48.917214 (+ 196us) write_op.cc:270] Start()
0113 08:54:48.917236 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459247108 usec, L: 0
0113 08:54:48.917454 (+ 218us) write_op.cc:270] Start()
0113 08:54:48.917476 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459253404 usec, L: 0
0113 08:54:48.917648 (+ 172us) write_op.cc:270] Start()
0113 08:54:48.917668 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459284535 usec, L: 0
0113 08:54:48.917851 (+ 183us) write_op.cc:270] Start()
0113 08:54:48.917901 (+ 50us) write_op.cc:276] Timestamp: P: 1736758459285502 usec, L: 0
0113 08:54:48.918085 (+ 184us) write_op.cc:270] Start()
0113 08:54:48.918127 (+ 42us) write_op.cc:276] Timestamp: P: 1736758459303821 usec, L: 0
0113 08:54:48.918307 (+ 180us) write_op.cc:270] Start()
0113 08:54:48.918327 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459309988 usec, L: 0
0113 08:54:48.918495 (+ 168us) write_op.cc:270] Start()
0113 08:54:48.918517 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459339945 usec, L: 0
0113 08:54:48.920684 (+ 2167us) write_op.cc:270] Start()
0113 08:54:48.920709 (+ 25us) write_op.cc:276] Timestamp: P: 1736758459355567 usec, L: 0
0113 08:54:48.920882 (+ 173us) write_op.cc:270] Start()
0113 08:54:48.920902 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459356929 usec, L: 0
0113 08:54:48.921076 (+ 174us) write_op.cc:270] Start()
0113 08:54:48.921098 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459417421 usec, L: 0
0113 08:54:48.921258 (+ 160us) write_op.cc:270] Start()
0113 08:54:48.921330 (+ 72us) write_op.cc:276] Timestamp: P: 1736758459425301 usec, L: 0
0113 08:54:48.921514 (+ 184us) write_op.cc:270] Start()
0113 08:54:48.921534 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459432378 usec, L: 0
0113 08:54:48.921707 (+ 173us) write_op.cc:270] Start()
0113 08:54:48.921727 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459433051 usec, L: 0
0113 08:54:48.921893 (+ 166us) write_op.cc:270] Start()
0113 08:54:48.921913 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459466084 usec, L: 0
0113 08:54:48.922094 (+ 181us) write_op.cc:270] Start()
0113 08:54:48.922140 (+ 46us) write_op.cc:276] Timestamp: P: 1736758459482110 usec, L: 0
0113 08:54:48.922762 (+ 622us) write_op.cc:270] Start()
0113 08:54:48.922785 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459484558 usec, L: 0
0113 08:54:48.923005 (+ 220us) write_op.cc:270] Start()
0113 08:54:48.923037 (+ 32us) write_op.cc:276] Timestamp: P: 1736758459500887 usec, L: 0
0113 08:54:48.923259 (+ 222us) write_op.cc:270] Start()
0113 08:54:48.923281 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459509790 usec, L: 0
0113 08:54:48.923449 (+ 168us) write_op.cc:270] Start()
0113 08:54:48.923469 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459546492 usec, L: 0
0113 08:54:48.923658 (+ 189us) write_op.cc:270] Start()
0113 08:54:48.923679 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459548077 usec, L: 0
0113 08:54:48.923827 (+ 148us) write_op.cc:270] Start()
0113 08:54:48.923846 (+ 19us) write_op.cc:276] Timestamp: P: 1736758459571847 usec, L: 0
0113 08:54:48.924050 (+ 204us) write_op.cc:270] Start()
0113 08:54:48.924074 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459584231 usec, L: 0
0113 08:54:48.924230 (+ 156us) write_op.cc:270] Start()
0113 08:54:48.924248 (+ 18us) write_op.cc:276] Timestamp: P: 1736758459594776 usec, L: 0
0113 08:54:48.924437 (+ 189us) write_op.cc:270] Start()
0113 08:54:48.924457 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459605461 usec, L: 0
0113 08:54:48.924607 (+ 150us) write_op.cc:270] Start()
0113 08:54:48.924625 (+ 18us) write_op.cc:276] Timestamp: P: 1736758459636991 usec, L: 0
0113 08:54:48.924823 (+ 198us) write_op.cc:270] Start()
0113 08:54:48.924844 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459644576 usec, L: 0
0113 08:54:48.924999 (+ 155us) write_op.cc:270] Start()
0113 08:54:48.925033 (+ 34us) write_op.cc:276] Timestamp: P: 1736758459651368 usec, L: 0
0113 08:54:48.929502 (+ 4469us) write_op.cc:270] Start()
0113 08:54:48.929529 (+ 27us) write_op.cc:276] Timestamp: P: 1736758459664449 usec, L: 0
0113 08:54:48.929712 (+ 183us) write_op.cc:270] Start()
0113 08:54:48.929779 (+ 67us) write_op.cc:276] Timestamp: P: 1736758459697935 usec, L: 0
0113 08:54:48.929989 (+ 210us) write_op.cc:270] Start()
0113 08:54:48.930009 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459713004 usec, L: 0
0113 08:54:48.930178 (+ 169us) write_op.cc:270] Start()
0113 08:54:48.930200 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459715093 usec, L: 0
0113 08:54:48.930402 (+ 202us) write_op.cc:270] Start()
0113 08:54:48.930422 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459720798 usec, L: 0
0113 08:54:48.930572 (+ 150us) write_op.cc:270] Start()
0113 08:54:48.930592 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459741003 usec, L: 0
0113 08:54:48.930838 (+ 246us) write_op.cc:270] Start()
0113 08:54:48.930860 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459786664 usec, L: 0
0113 08:54:48.931061 (+ 201us) write_op.cc:270] Start()
0113 08:54:48.931085 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459789691 usec, L: 0
0113 08:54:48.931258 (+ 173us) write_op.cc:270] Start()
0113 08:54:48.931287 (+ 29us) write_op.cc:276] Timestamp: P: 1736758459792111 usec, L: 0
0113 08:54:48.931467 (+ 180us) write_op.cc:270] Start()
0113 08:54:48.931487 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459800444 usec, L: 0
0113 08:54:48.931663 (+ 176us) write_op.cc:270] Start()
0113 08:54:48.931683 (+ 20us) write_op.cc:276] Timestamp: P: 1736758459844236 usec, L: 0
0113 08:54:48.931897 (+ 214us) write_op.cc:270] Start()
0113 08:54:48.931919 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459844958 usec, L: 0
0113 08:54:48.932396 (+ 477us) write_op.cc:270] Start()
0113 08:54:48.932418 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459870494 usec, L: 0
0113 08:54:48.932638 (+ 220us) write_op.cc:270] Start()
0113 08:54:48.932660 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459871296 usec, L: 0
0113 08:54:48.932842 (+ 182us) write_op.cc:270] Start()
0113 08:54:48.932865 (+ 23us) write_op.cc:276] Timestamp: P: 1736758459904396 usec, L: 0
0113 08:54:48.933066 (+ 201us) write_op.cc:270] Start()
0113 08:54:48.933087 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459919657 usec, L: 0
0113 08:54:48.933258 (+ 171us) write_op.cc:270] Start()
0113 08:54:48.933279 (+ 21us) write_op.cc:276] Timestamp: P: 1736758459935817 usec, L: 0
0113 08:54:48.934566 (+ 1287us) write_op.cc:270] Start()
0113 08:54:48.934590 (+ 24us) write_op.cc:276] Timestamp: P: 1736758459948298 usec, L: 0
0113 08:54:48.934788 (+ 198us) write_op.cc:270] Start()
0113 08:54:48.934833 (+ 45us) write_op.cc:276] Timestamp: P: 1736758459977419 usec, L: 0
0113 08:54:48.935049 (+ 216us) write_op.cc:270] Start()
0113 08:54:48.935096 (+ 47us) write_op.cc:276] Timestamp: P: 1736758459983596 usec, L: 0
0113 08:54:48.935299 (+ 203us) write_op.cc:270] Start()
0113 08:54:48.935321 (+ 22us) write_op.cc:276] Timestamp: P: 1736758459989640 usec, L: 0
0113 08:54:48.935532 (+ 211us) write_op.cc:270] Start()
0113 08:54:48.935552 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460021753 usec, L: 0
0113 08:54:48.935727 (+ 175us) write_op.cc:270] Start()
0113 08:54:48.935748 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460043861 usec, L: 0
0113 08:54:48.935927 (+ 179us) write_op.cc:270] Start()
0113 08:54:48.935948 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460044882 usec, L: 0
0113 08:54:48.936126 (+ 178us) write_op.cc:270] Start()
0113 08:54:48.936179 (+ 53us) write_op.cc:276] Timestamp: P: 1736758460062101 usec, L: 0
0113 08:54:48.936358 (+ 179us) write_op.cc:270] Start()
0113 08:54:48.936379 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460065050 usec, L: 0
0113 08:54:48.936567 (+ 188us) write_op.cc:270] Start()
0113 08:54:48.936588 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460100330 usec, L: 0
0113 08:54:48.936779 (+ 191us) write_op.cc:270] Start()
0113 08:54:48.936801 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460101091 usec, L: 0
0113 08:54:48.936983 (+ 182us) write_op.cc:270] Start()
0113 08:54:48.937003 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460128999 usec, L: 0
0113 08:54:48.937175 (+ 172us) write_op.cc:270] Start()
0113 08:54:48.937196 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460130081 usec, L: 0
0113 08:54:48.937365 (+ 169us) write_op.cc:270] Start()
0113 08:54:48.937386 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460154284 usec, L: 0
0113 08:54:48.937556 (+ 170us) write_op.cc:270] Start()
0113 08:54:48.937577 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460165351 usec, L: 0
0113 08:54:48.937793 (+ 216us) write_op.cc:270] Start()
0113 08:54:48.937815 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460182338 usec, L: 0
0113 08:54:48.937987 (+ 172us) write_op.cc:270] Start()
0113 08:54:48.938008 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460183284 usec, L: 0
0113 08:54:48.938183 (+ 175us) write_op.cc:270] Start()
0113 08:54:48.938203 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460216667 usec, L: 0
0113 08:54:48.938376 (+ 173us) write_op.cc:270] Start()
0113 08:54:48.938398 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460231974 usec, L: 0
0113 08:54:48.938571 (+ 173us) write_op.cc:270] Start()
0113 08:54:48.938591 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460246989 usec, L: 0
0113 08:54:48.946719 (+ 8128us) write_op.cc:270] Start()
0113 08:54:48.946790 (+ 71us) write_op.cc:276] Timestamp: P: 1736758460248827 usec, L: 0
0113 08:54:48.947167 (+ 377us) write_op.cc:270] Start()
0113 08:54:48.947193 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460284057 usec, L: 0
0113 08:54:48.947424 (+ 231us) write_op.cc:270] Start()
0113 08:54:48.947446 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460292983 usec, L: 0
0113 08:54:48.947632 (+ 186us) write_op.cc:270] Start()
0113 08:54:48.947668 (+ 36us) write_op.cc:276] Timestamp: P: 1736758460311411 usec, L: 0
0113 08:54:48.947867 (+ 199us) write_op.cc:270] Start()
0113 08:54:48.947889 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460336856 usec, L: 0
0113 08:54:48.948077 (+ 188us) write_op.cc:270] Start()
0113 08:54:48.948099 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460338615 usec, L: 0
0113 08:54:48.948279 (+ 180us) write_op.cc:270] Start()
0113 08:54:48.948298 (+ 19us) write_op.cc:276] Timestamp: P: 1736758460341506 usec, L: 0
0113 08:54:48.948475 (+ 177us) write_op.cc:270] Start()
0113 08:54:48.948495 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460347784 usec, L: 0
0113 08:54:48.948667 (+ 172us) write_op.cc:270] Start()
0113 08:54:48.948687 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460395728 usec, L: 0
0113 08:54:48.948883 (+ 196us) write_op.cc:270] Start()
0113 08:54:48.948907 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460408221 usec, L: 0
0113 08:54:48.949098 (+ 191us) write_op.cc:270] Start()
0113 08:54:48.949120 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460408960 usec, L: 0
0113 08:54:48.949305 (+ 185us) write_op.cc:270] Start()
0113 08:54:48.949326 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460413826 usec, L: 0
0113 08:54:48.949637 (+ 311us) write_op.cc:270] Start()
0113 08:54:48.949659 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460457762 usec, L: 0
0113 08:54:48.949861 (+ 202us) write_op.cc:270] Start()
0113 08:54:48.949885 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460476948 usec, L: 0
0113 08:54:48.950088 (+ 203us) write_op.cc:270] Start()
0113 08:54:48.950579 (+ 491us) write_op.cc:276] Timestamp: P: 1736758460502571 usec, L: 0
0113 08:54:48.950798 (+ 219us) write_op.cc:270] Start()
0113 08:54:48.950887 (+ 89us) write_op.cc:276] Timestamp: P: 1736758460503936 usec, L: 0
0113 08:54:48.951112 (+ 225us) write_op.cc:270] Start()
0113 08:54:48.951137 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460531350 usec, L: 0
0113 08:54:48.951332 (+ 195us) write_op.cc:270] Start()
0113 08:54:48.951354 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460555031 usec, L: 0
0113 08:54:48.951916 (+ 562us) write_op.cc:270] Start()
0113 08:54:48.951936 (+ 20us) write_op.cc:276] Timestamp: P: 1736758460564990 usec, L: 0
0113 08:54:48.952553 (+ 617us) write_op.cc:270] Start()
0113 08:54:48.952579 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460584221 usec, L: 0
0113 08:54:48.952789 (+ 210us) write_op.cc:270] Start()
0113 08:54:48.952811 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460606315 usec, L: 0
0113 08:54:48.954500 (+ 1689us) write_op.cc:270] Start()
0113 08:54:48.954531 (+ 31us) write_op.cc:276] Timestamp: P: 1736758460607726 usec, L: 0
0113 08:54:48.954748 (+ 217us) write_op.cc:270] Start()
0113 08:54:48.954781 (+ 33us) write_op.cc:276] Timestamp: P: 1736758460622277 usec, L: 0
0113 08:54:48.955030 (+ 249us) write_op.cc:270] Start()
0113 08:54:48.955053 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460625291 usec, L: 0
0113 08:54:48.955245 (+ 192us) write_op.cc:270] Start()
0113 08:54:48.955277 (+ 32us) write_op.cc:276] Timestamp: P: 1736758460651234 usec, L: 0
0113 08:54:48.955520 (+ 243us) write_op.cc:270] Start()
0113 08:54:48.955542 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460680385 usec, L: 0
0113 08:54:48.955729 (+ 187us) write_op.cc:270] Start()
0113 08:54:48.955752 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460700580 usec, L: 0
0113 08:54:48.960304 (+ 4552us) write_op.cc:270] Start()
0113 08:54:48.960338 (+ 34us) write_op.cc:276] Timestamp: P: 1736758460704378 usec, L: 0
0113 08:54:48.960601 (+ 263us) write_op.cc:270] Start()
0113 08:54:48.960624 (+ 23us) write_op.cc:276] Timestamp: P: 1736758460712914 usec, L: 0
0113 08:54:48.960838 (+ 214us) write_op.cc:270] Start()
0113 08:54:48.960860 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460721445 usec, L: 0
0113 08:54:48.961118 (+ 258us) write_op.cc:270] Start()
0113 08:54:48.961158 (+ 40us) write_op.cc:276] Timestamp: P: 1736758460768240 usec, L: 0
0113 08:54:48.961418 (+ 260us) write_op.cc:270] Start()
0113 08:54:48.961442 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460770681 usec, L: 0
0113 08:54:48.961831 (+ 389us) write_op.cc:270] Start()
0113 08:54:48.961856 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460771487 usec, L: 0
0113 08:54:48.962098 (+ 242us) write_op.cc:270] Start()
0113 08:54:48.962124 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460789494 usec, L: 0
0113 08:54:48.962333 (+ 209us) write_op.cc:270] Start()
0113 08:54:48.962355 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460820966 usec, L: 0
0113 08:54:48.962555 (+ 200us) write_op.cc:270] Start()
0113 08:54:48.962668 (+ 113us) write_op.cc:276] Timestamp: P: 1736758460822189 usec, L: 0
0113 08:54:48.962938 (+ 270us) write_op.cc:270] Start()
0113 08:54:48.962962 (+ 24us) write_op.cc:276] Timestamp: P: 1736758460824187 usec, L: 0
0113 08:54:48.963378 (+ 416us) write_op.cc:270] Start()
0113 08:54:48.963403 (+ 25us) write_op.cc:276] Timestamp: P: 1736758460862907 usec, L: 0
0113 08:54:48.963592 (+ 189us) write_op.cc:270] Start()
0113 08:54:48.963613 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460881654 usec, L: 0
0113 08:54:48.963813 (+ 200us) write_op.cc:270] Start()
0113 08:54:48.963834 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460884305 usec, L: 0
0113 08:54:48.964019 (+ 185us) write_op.cc:270] Start()
0113 08:54:48.964052 (+ 33us) write_op.cc:276] Timestamp: P: 1736758460903343 usec, L: 0
0113 08:54:48.964249 (+ 197us) write_op.cc:270] Start()
0113 08:54:48.964271 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460928652 usec, L: 0
0113 08:54:48.964574 (+ 303us) write_op.cc:270] Start()
0113 08:54:48.964600 (+ 26us) write_op.cc:276] Timestamp: P: 1736758460939585 usec, L: 0
0113 08:54:48.965119 (+ 519us) write_op.cc:270] Start()
0113 08:54:48.965141 (+ 22us) write_op.cc:276] Timestamp: P: 1736758460946301 usec, L: 0
0113 08:54:48.965347 (+ 206us) write_op.cc:270] Start()
0113 08:54:48.965368 (+ 21us) write_op.cc:276] Timestamp: P: 1736758460962383 usec, L: 0
0113 08:54:48.967376 (+ 2008us) write_op.cc:270] Start()
0113 08:54:48.967429 (+ 53us) write_op.cc:276] Timestamp: P: 1736758461014950 usec, L: 0
0113 08:54:48.973071 (+ 5642us) write_op.cc:270] Start()
0113 08:54:48.973109 (+ 38us) write_op.cc:276] Timestamp: P: 1736758461016479 usec, L: 0
0113 08:54:48.973915 (+ 806us) write_op.cc:270] Start()
0113 08:54:48.973949 (+ 34us) write_op.cc:276] Timestamp: P: 1736758461017328 usec, L: 0
0113 08:54:48.975954 (+ 2005us) write_op.cc:270] Start()
0113 08:54:48.975992 (+ 38us) write_op.cc:276] Timestamp: P: 1736758461020181 usec, L: 0
0113 08:54:48.978512 (+ 2520us) write_op.cc:270] Start()
0113 08:54:48.978543 (+ 31us) write_op.cc:276] Timestamp: P: 1736758461064097 usec, L: 0
0113 08:54:48.978780 (+ 237us) write_op.cc:270] Start()
0113 08:54:48.978800 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461064850 usec, L: 0
0113 08:54:48.979029 (+ 229us) write_op.cc:270] Start()
0113 08:54:48.979049 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461094524 usec, L: 0
0113 08:54:48.979204 (+ 155us) write_op.cc:270] Start()
0113 08:54:48.979221 (+ 17us) write_op.cc:276] Timestamp: P: 1736758461096257 usec, L: 0
0113 08:54:48.979422 (+ 201us) write_op.cc:270] Start()
0113 08:54:48.979440 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461113733 usec, L: 0
0113 08:54:48.979598 (+ 158us) write_op.cc:270] Start()
0113 08:54:48.979616 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461118318 usec, L: 0
0113 08:54:48.980925 (+ 1309us) write_op.cc:270] Start()
0113 08:54:48.980954 (+ 29us) write_op.cc:276] Timestamp: P: 1736758461158472 usec, L: 0
0113 08:54:48.981140 (+ 186us) write_op.cc:270] Start()
0113 08:54:48.981158 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461166249 usec, L: 0
0113 08:54:48.981302 (+ 144us) write_op.cc:270] Start()
0113 08:54:48.981320 (+ 18us) write_op.cc:276] Timestamp: P: 1736758461172275 usec, L: 0
0113 08:54:48.983011 (+ 1691us) write_op.cc:270] Start()
0113 08:54:48.983036 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461182030 usec, L: 0
0113 08:54:48.983219 (+ 183us) write_op.cc:270] Start()
0113 08:54:48.983240 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461228985 usec, L: 0
0113 08:54:48.983456 (+ 216us) write_op.cc:270] Start()
0113 08:54:48.983478 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461237507 usec, L: 0
0113 08:54:48.983630 (+ 152us) write_op.cc:270] Start()
0113 08:54:48.983652 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461238268 usec, L: 0
0113 08:54:48.983870 (+ 218us) write_op.cc:270] Start()
0113 08:54:48.983893 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461238965 usec, L: 0
0113 08:54:48.984077 (+ 184us) write_op.cc:270] Start()
0113 08:54:48.984101 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461280724 usec, L: 0
0113 08:54:48.987039 (+ 2938us) write_op.cc:270] Start()
0113 08:54:48.987065 (+ 26us) write_op.cc:276] Timestamp: P: 1736758461282533 usec, L: 0
0113 08:54:48.987276 (+ 211us) write_op.cc:270] Start()
0113 08:54:48.987297 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461288285 usec, L: 0
0113 08:54:48.987475 (+ 178us) write_op.cc:270] Start()
0113 08:54:48.987496 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461296567 usec, L: 0
0113 08:54:48.987752 (+ 256us) write_op.cc:270] Start()
0113 08:54:48.987774 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461337156 usec, L: 0
0113 08:54:48.987989 (+ 215us) write_op.cc:270] Start()
0113 08:54:48.988013 (+ 24us) write_op.cc:276] Timestamp: P: 1736758461338051 usec, L: 0
0113 08:54:48.988230 (+ 217us) write_op.cc:270] Start()
0113 08:54:48.988253 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461341779 usec, L: 0
0113 08:54:48.988441 (+ 188us) write_op.cc:270] Start()
0113 08:54:48.988511 (+ 70us) write_op.cc:276] Timestamp: P: 1736758461359229 usec, L: 0
0113 08:54:48.988927 (+ 416us) write_op.cc:270] Start()
0113 08:54:48.988950 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461398909 usec, L: 0
0113 08:54:48.989331 (+ 381us) write_op.cc:270] Start()
0113 08:54:48.989354 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461401039 usec, L: 0
0113 08:54:48.994631 (+ 5277us) write_op.cc:270] Start()
0113 08:54:48.994670 (+ 39us) write_op.cc:276] Timestamp: P: 1736758461405066 usec, L: 0
0113 08:54:48.995198 (+ 528us) write_op.cc:270] Start()
0113 08:54:48.995226 (+ 28us) write_op.cc:276] Timestamp: P: 1736758461429771 usec, L: 0
0113 08:54:48.995415 (+ 189us) write_op.cc:270] Start()
0113 08:54:48.995436 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461467579 usec, L: 0
0113 08:54:48.995653 (+ 217us) write_op.cc:270] Start()
0113 08:54:48.995673 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461484541 usec, L: 0
0113 08:54:48.995873 (+ 200us) write_op.cc:270] Start()
0113 08:54:48.995895 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461486658 usec, L: 0
0113 08:54:48.996095 (+ 200us) write_op.cc:270] Start()
0113 08:54:48.996117 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461499007 usec, L: 0
0113 08:54:48.996363 (+ 246us) write_op.cc:270] Start()
0113 08:54:48.996386 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461540110 usec, L: 0
0113 08:54:48.996685 (+ 299us) write_op.cc:270] Start()
0113 08:54:48.996708 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461566090 usec, L: 0
0113 08:54:48.996970 (+ 262us) write_op.cc:270] Start()
0113 08:54:48.996993 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461567239 usec, L: 0
0113 08:54:48.997193 (+ 200us) write_op.cc:270] Start()
0113 08:54:48.997216 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461575770 usec, L: 0
0113 08:54:48.997418 (+ 202us) write_op.cc:270] Start()
0113 08:54:48.997437 (+ 19us) write_op.cc:276] Timestamp: P: 1736758461605669 usec, L: 0
0113 08:54:48.997653 (+ 216us) write_op.cc:270] Start()
0113 08:54:48.997675 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461633599 usec, L: 0
0113 08:54:48.998289 (+ 614us) write_op.cc:270] Start()
0113 08:54:48.998312 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461636595 usec, L: 0
0113 08:54:49.001182 (+ 2870us) write_op.cc:270] Start()
0113 08:54:49.001218 (+ 36us) write_op.cc:276] Timestamp: P: 1736758461650266 usec, L: 0
0113 08:54:49.008616 (+ 7398us) write_op.cc:270] Start()
0113 08:54:49.008652 (+ 36us) write_op.cc:276] Timestamp: P: 1736758461671830 usec, L: 0
0113 08:54:49.008947 (+ 295us) write_op.cc:270] Start()
0113 08:54:49.008972 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461700621 usec, L: 0
0113 08:54:49.009467 (+ 495us) write_op.cc:270] Start()
0113 08:54:49.009488 (+ 21us) write_op.cc:276] Timestamp: P: 1736758461707341 usec, L: 0
0113 08:54:49.015228 (+ 5740us) write_op.cc:270] Start()
0113 08:54:49.015272 (+ 44us) write_op.cc:276] Timestamp: P: 1736758461708871 usec, L: 0
0113 08:54:49.028566 (+ 13294us) write_op.cc:270] Start()
0113 08:54:49.028611 (+ 45us) write_op.cc:276] Timestamp: P: 1736758461738843 usec, L: 0
0113 08:54:49.028912 (+ 301us) write_op.cc:270] Start()
0113 08:54:49.028939 (+ 27us) write_op.cc:276] Timestamp: P: 1736758461764766 usec, L: 0
0113 08:54:49.029137 (+ 198us) write_op.cc:270] Start()
0113 08:54:49.029160 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461791798 usec, L: 0
0113 08:54:49.029360 (+ 200us) write_op.cc:270] Start()
0113 08:54:49.029383 (+ 23us) write_op.cc:276] Timestamp: P: 1736758461795244 usec, L: 0
0113 08:54:49.029547 (+ 164us) write_op.cc:270] Start()
0113 08:54:49.029569 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461825090 usec, L: 0
0113 08:54:49.029735 (+ 166us) write_op.cc:270] Start()
0113 08:54:49.029757 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461846401 usec, L: 0
0113 08:54:49.029917 (+ 160us) write_op.cc:270] Start()
0113 08:54:49.029937 (+ 20us) write_op.cc:276] Timestamp: P: 1736758461884824 usec, L: 0
0113 08:54:49.030229 (+ 292us) write_op.cc:270] Start()
0113 08:54:49.030256 (+ 27us) write_op.cc:276] Timestamp: P: 1736758461901915 usec, L: 0
0113 08:54:49.030421 (+ 165us) write_op.cc:270] Start()
0113 08:54:49.030443 (+ 22us) write_op.cc:276] Timestamp: P: 1736758461913657 usec, L: 0
0113 08:54:49.030631 (+ 188us) write_op.cc:270] Start()
0113 08:54:49.030656 (+ 25us) write_op.cc:276] Timestamp: P: 1736758461924539 usec, L: 0
0113 08:54:49.030873 (+ 217us) write_op.cc:270] Start()
0113 08:54:49.030896 (+ 23us) write_op.cc:276] Timestamp: P: 1736
I20250113 08:54:52.874467 8496 ts_manager.cc:281] Unset tserver state for aea19a9cd21144508fdcfd7b3570a0e1 from MAINTENANCE_MODE
I20250113 08:54:53.206869 8147 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:53.254618 8711 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:53.272665 8016 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:54:53.645390 8406 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
W20250113 08:54:54.517889 8234 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.000s sys 0.000s
W20250113 08:54:55.990685 8037 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.123.1:46439, user_credentials={real_user=slave}} blocked reactor thread for 61431us
I20250113 08:54:56.348173 8496 ts_manager.cc:292] Set tserver state for aea19a9cd21144508fdcfd7b3570a0e1 to MAINTENANCE_MODE
I20250113 08:54:56.351884 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 8562
I20250113 08:54:56.354039 8706 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:54:56.962648 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 8562
W20250113 08:54:57.086597 8037 connection.cc:537] client connection to 127.7.123.1:46439 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:54:57.086926 7906 connection.cc:537] client connection to 127.7.123.1:46439 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:54:57.087039 7906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 26 similar messages]
W20250113 08:54:57.086769 8037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 53 similar messages]
W20250113 08:54:57.112444 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:57.112807 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:57.113003 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:57.113178 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:57.118378 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:57.118667 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:57.522411 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:57.545819 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:57.594087 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:57.624876 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:57.642517 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:57.686035 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:57.964938 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:58.013919 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:58.096518 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:58.148363 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:58.193791 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:54:58.241161 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250113 08:54:58.305186 8695 consensus_queue.cc:579] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.028s)
I20250113 08:54:58.358142 8830 consensus_queue.cc:579] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.049s)
I20250113 08:54:58.410499 8823 consensus_queue.cc:579] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.079s)
I20250113 08:54:58.426892 8830 consensus_queue.cc:579] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.071s)
I20250113 08:54:58.472645 8822 consensus_queue.cc:579] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.117s)
W20250113 08:54:58.477054 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:58.500667 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:58.654870 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:58.695916 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:58.734493 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:58.764745 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:58.961282 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250113 08:54:58.981904 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250113 08:54:59.029799 8828 consensus_queue.cc:579] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.080s)
W20250113 08:54:59.158033 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250113 08:54:59.212563 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250113 08:54:59.286209 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250113 08:54:59.362282 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250113 08:54:59.493981 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:59.529989 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:59.645747 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:59.709005 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:59.827237 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:54:59.917447 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250113 08:55:00.024430 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20250113 08:55:00.071020 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20250113 08:55:00.082914 8493 ts_manager.cc:281] Unset tserver state for aea19a9cd21144508fdcfd7b3570a0e1 from MAINTENANCE_MODE
I20250113 08:55:00.083928 8147 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:55:00.129117 8099 consensus_queue.cc:237] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2072, Committed index: 2072, Last appended: 2.2072, Last appended by leader: 201, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:00.129503 8101 consensus_queue.cc:237] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2070, Committed index: 2070, Last appended: 1.2071, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2072 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:00.129117 8100 consensus_queue.cc:237] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2072, Committed index: 2072, Last appended: 2.2072, Last appended by leader: 202, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:00.135005 7970 raft_consensus.cc:1270] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Refusing update from remote peer 81f9c074161846b1b2a552cb12d3ef30: Log matching property violated. Preceding OpId in replica: term: 2 index: 2072. Preceding OpId from leader: term: 2 index: 2073. (index mismatch)
I20250113 08:55:00.135035 7967 raft_consensus.cc:1270] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Refusing update from remote peer 81f9c074161846b1b2a552cb12d3ef30: Log matching property violated. Preceding OpId in replica: term: 2 index: 2072. Preceding OpId from leader: term: 2 index: 2073. (index mismatch)
I20250113 08:55:00.136502 8828 consensus_queue.cc:1035] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2073, Last known committed idx: 2072, Time since last communication: 0.000s
I20250113 08:55:00.129524 8097 consensus_queue.cc:237] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2072, Committed index: 2072, Last appended: 2.2072, Last appended by leader: 201, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:00.147321 8828 consensus_queue.cc:1035] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2073, Last known committed idx: 2072, Time since last communication: 0.000s
I20250113 08:55:00.148774 7970 raft_consensus.cc:1270] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Refusing update from remote peer 81f9c074161846b1b2a552cb12d3ef30: Log matching property violated. Preceding OpId in replica: term: 2 index: 2072. Preceding OpId from leader: term: 2 index: 2073. (index mismatch)
I20250113 08:55:00.149446 7970 raft_consensus.cc:1270] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Refusing update from remote peer 81f9c074161846b1b2a552cb12d3ef30: Log matching property violated. Preceding OpId in replica: term: 1 index: 2071. Preceding OpId from leader: term: 1 index: 2072. (index mismatch)
I20250113 08:55:00.150435 8817 consensus_queue.cc:1035] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2073, Last known committed idx: 2072, Time since last communication: 0.000s
I20250113 08:55:00.150604 8828 consensus_queue.cc:1035] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2072, Last known committed idx: 2070, Time since last communication: 0.000s
I20250113 08:55:00.154973 8817 raft_consensus.cc:2949] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 LEADER]: Committing config change with OpId 2.2073: config changed from index -1 to 2073, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.158546 8818 raft_consensus.cc:2949] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 LEADER]: Committing config change with OpId 2.2073: config changed from index -1 to 2073, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.158421 7970 raft_consensus.cc:2949] T fe2a45627bfe4db0b9cdccfab81e2805 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Committing config change with OpId 2.2073: config changed from index -1 to 2073, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.161221 8804 raft_consensus.cc:2949] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [term 1 LEADER]: Committing config change with OpId 1.2072: config changed from index -1 to 2072, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2072 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.163242 8481 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 44c2ecb48ca14049a051ee94f7d58cd6 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250113 08:55:00.164664 7969 raft_consensus.cc:2949] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Committing config change with OpId 2.2073: config changed from index -1 to 2073, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.166940 7971 raft_consensus.cc:2949] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 [term 1 FOLLOWER]: Committing config change with OpId 1.2072: config changed from index -1 to 2072, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2072 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.172917 8481 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe2a45627bfe4db0b9cdccfab81e2805 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250113 08:55:00.173619 8481 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c1e64a047ae649ac9afc1b77608c067f with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250113 08:55:00.179030 8494 catalog_manager.cc:5526] T 44c2ecb48ca14049a051ee94f7d58cd6 P 1abc86bb580b4d14876623243c9d6594 reported cstate change: config changed from index -1 to 2073, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New cstate: current_term: 2 leader_uuid: "81f9c074161846b1b2a552cb12d3ef30" committed_config { opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.180248 8494 catalog_manager.cc:5526] T c1e64a047ae649ac9afc1b77608c067f P 1abc86bb580b4d14876623243c9d6594 reported cstate change: config changed from index -1 to 2072, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New cstate: current_term: 1 leader_uuid: "81f9c074161846b1b2a552cb12d3ef30" committed_config { opid_index: 2072 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
W20250113 08:55:00.188213 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:55:00.187826 8493 catalog_manager.cc:5526] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 reported cstate change: config changed from index -1 to 2073, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New cstate: current_term: 2 leader_uuid: "81f9c074161846b1b2a552cb12d3ef30" committed_config { opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250113 08:55:00.188336 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:00.188414 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:00.188588 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:55:00.192294 8827 raft_consensus.cc:2949] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [term 2 LEADER]: Committing config change with OpId 2.2073: config changed from index -1 to 2073, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.193142 8016 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
I20250113 08:55:00.200716 8481 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 34cc82c767a44ffab6786f89b8d41a00 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250113 08:55:00.207758 7971 raft_consensus.cc:2949] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Committing config change with OpId 2.2073: config changed from index -1 to 2073, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.209306 8493 catalog_manager.cc:5526] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 reported cstate change: config changed from index -1 to 2073, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New cstate: current_term: 2 leader_uuid: "81f9c074161846b1b2a552cb12d3ef30" committed_config { opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250113 08:55:00.218963 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe2a45627bfe4db0b9cdccfab81e2805. This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:00.219233 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44c2ecb48ca14049a051ee94f7d58cd6. This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:00.219430 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34cc82c767a44ffab6786f89b8d41a00. This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:00.219615 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1e64a047ae649ac9afc1b77608c067f. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:55:00.241048 8861 ts_tablet_manager.cc:927] T c1e64a047ae649ac9afc1b77608c067f P 7eb1620e70d84c44b17b3e0aad7c38bc: Initiating tablet copy from peer 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:55:00.246635 7970 consensus_queue.cc:237] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2075, Committed index: 2075, Last appended: 1.2076, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:00.249903 7971 consensus_queue.cc:237] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2075, Committed index: 2075, Last appended: 1.2076, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:00.252630 8863 ts_tablet_manager.cc:927] T 44c2ecb48ca14049a051ee94f7d58cd6 P 7eb1620e70d84c44b17b3e0aad7c38bc: Initiating tablet copy from peer 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:55:00.254551 8861 tablet_copy_client.cc:323] T c1e64a047ae649ac9afc1b77608c067f P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.123.3:46625
I20250113 08:55:00.257462 8097 raft_consensus.cc:1270] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Refusing update from remote peer 1abc86bb580b4d14876623243c9d6594: Log matching property violated. Preceding OpId in replica: term: 1 index: 2076. Preceding OpId from leader: term: 1 index: 2077. (index mismatch)
I20250113 08:55:00.261652 8863 tablet_copy_client.cc:323] T 44c2ecb48ca14049a051ee94f7d58cd6 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.123.3:46625
I20250113 08:55:00.264621 8798 consensus_queue.cc:1035] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2077, Last known committed idx: 2075, Time since last communication: 0.000s
I20250113 08:55:00.272364 8101 raft_consensus.cc:1270] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Refusing update from remote peer 1abc86bb580b4d14876623243c9d6594: Log matching property violated. Preceding OpId in replica: term: 1 index: 2076. Preceding OpId from leader: term: 1 index: 2077. (index mismatch)
I20250113 08:55:00.278281 8849 consensus_queue.cc:1035] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2077, Last known committed idx: 2075, Time since last communication: 0.000s
I20250113 08:55:00.286762 8798 raft_consensus.cc:2949] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 [term 1 LEADER]: Committing config change with OpId 1.2077: config changed from index -1 to 2077, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.280244 8862 ts_tablet_manager.cc:927] T fe2a45627bfe4db0b9cdccfab81e2805 P 7eb1620e70d84c44b17b3e0aad7c38bc: Initiating tablet copy from peer 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:55:00.295560 8483 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c35e4ecbe2cc424089eaa5e0249df483 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250113 08:55:00.296334 8869 ts_tablet_manager.cc:927] T 34cc82c767a44ffab6786f89b8d41a00 P 7eb1620e70d84c44b17b3e0aad7c38bc: Initiating tablet copy from peer 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
I20250113 08:55:00.297109 8494 catalog_manager.cc:5526] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 reported cstate change: config changed from index -1 to 2077, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New cstate: current_term: 1 leader_uuid: "1abc86bb580b4d14876623243c9d6594" committed_config { opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:55:00.304497 8101 raft_consensus.cc:2949] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Committing config change with OpId 1.2077: config changed from index -1 to 2077, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.314230 8798 raft_consensus.cc:2949] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 [term 1 LEADER]: Committing config change with OpId 1.2077: config changed from index -1 to 2077, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.319077 8483 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 993904453c594e2db077d13f62d9b27a with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250113 08:55:00.320183 8862 tablet_copy_client.cc:323] T fe2a45627bfe4db0b9cdccfab81e2805 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.123.3:46625
I20250113 08:55:00.325006 8494 catalog_manager.cc:5526] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 reported cstate change: config changed from index -1 to 2077, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New cstate: current_term: 1 leader_uuid: "1abc86bb580b4d14876623243c9d6594" committed_config { opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250113 08:55:00.333667 8869 tablet_copy_client.cc:323] T 34cc82c767a44ffab6786f89b8d41a00 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.123.3:46625
I20250113 08:55:00.334471 8122 tablet_copy_service.cc:140] P 81f9c074161846b1b2a552cb12d3ef30: Received BeginTabletCopySession request for tablet c1e64a047ae649ac9afc1b77608c067f from peer 7eb1620e70d84c44b17b3e0aad7c38bc ({username='slave'} at 127.7.123.4:48611)
I20250113 08:55:00.334890 8122 tablet_copy_service.cc:161] P 81f9c074161846b1b2a552cb12d3ef30: Beginning new tablet copy session on tablet c1e64a047ae649ac9afc1b77608c067f from peer 7eb1620e70d84c44b17b3e0aad7c38bc at {username='slave'} at 127.7.123.4:48611: session id = 7eb1620e70d84c44b17b3e0aad7c38bc-c1e64a047ae649ac9afc1b77608c067f
I20250113 08:55:00.339385 8097 raft_consensus.cc:2949] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Committing config change with OpId 1.2077: config changed from index -1 to 2077, NON_VOTER 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4) added. New config: { opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } } }
I20250113 08:55:00.342984 8120 tablet_copy_service.cc:140] P 81f9c074161846b1b2a552cb12d3ef30: Received BeginTabletCopySession request for tablet fe2a45627bfe4db0b9cdccfab81e2805 from peer 7eb1620e70d84c44b17b3e0aad7c38bc ({username='slave'} at 127.7.123.4:48611)
I20250113 08:55:00.343250 8120 tablet_copy_service.cc:161] P 81f9c074161846b1b2a552cb12d3ef30: Beginning new tablet copy session on tablet fe2a45627bfe4db0b9cdccfab81e2805 from peer 7eb1620e70d84c44b17b3e0aad7c38bc at {username='slave'} at 127.7.123.4:48611: session id = 7eb1620e70d84c44b17b3e0aad7c38bc-fe2a45627bfe4db0b9cdccfab81e2805
W20250113 08:55:00.348594 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:00.348765 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:00.349642 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c35e4ecbe2cc424089eaa5e0249df483. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:55:00.334457 8121 tablet_copy_service.cc:140] P 81f9c074161846b1b2a552cb12d3ef30: Received BeginTabletCopySession request for tablet 44c2ecb48ca14049a051ee94f7d58cd6 from peer 7eb1620e70d84c44b17b3e0aad7c38bc ({username='slave'} at 127.7.123.4:48611)
I20250113 08:55:00.349900 8121 tablet_copy_service.cc:161] P 81f9c074161846b1b2a552cb12d3ef30: Beginning new tablet copy session on tablet 44c2ecb48ca14049a051ee94f7d58cd6 from peer 7eb1620e70d84c44b17b3e0aad7c38bc at {username='slave'} at 127.7.123.4:48611: session id = 7eb1620e70d84c44b17b3e0aad7c38bc-44c2ecb48ca14049a051ee94f7d58cd6
W20250113 08:55:00.349965 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 993904453c594e2db077d13f62d9b27a. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:55:00.353202 8121 tablet_copy_source_session.cc:215] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:55:00.355114 8120 tablet_copy_source_session.cc:215] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:55:00.356689 8122 tablet_copy_source_session.cc:215] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:55:00.380560 8861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e64a047ae649ac9afc1b77608c067f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:55:00.383064 8862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe2a45627bfe4db0b9cdccfab81e2805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:55:00.384791 8863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c2ecb48ca14049a051ee94f7d58cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:55:00.386688 8122 tablet_copy_service.cc:140] P 81f9c074161846b1b2a552cb12d3ef30: Received BeginTabletCopySession request for tablet 34cc82c767a44ffab6786f89b8d41a00 from peer 7eb1620e70d84c44b17b3e0aad7c38bc ({username='slave'} at 127.7.123.4:48611)
I20250113 08:55:00.390057 8122 tablet_copy_service.cc:161] P 81f9c074161846b1b2a552cb12d3ef30: Beginning new tablet copy session on tablet 34cc82c767a44ffab6786f89b8d41a00 from peer 7eb1620e70d84c44b17b3e0aad7c38bc at {username='slave'} at 127.7.123.4:48611: session id = 7eb1620e70d84c44b17b3e0aad7c38bc-34cc82c767a44ffab6786f89b8d41a00
I20250113 08:55:00.393245 8122 tablet_copy_source_session.cc:215] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:55:00.402732 8869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34cc82c767a44ffab6786f89b8d41a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:55:00.414356 8872 ts_tablet_manager.cc:927] T 993904453c594e2db077d13f62d9b27a P 7eb1620e70d84c44b17b3e0aad7c38bc: Initiating tablet copy from peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:55:00.428323 8869 tablet_copy_client.cc:806] T 34cc82c767a44ffab6786f89b8d41a00 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 0 data blocks...
I20250113 08:55:00.429306 8869 tablet_copy_client.cc:670] T 34cc82c767a44ffab6786f89b8d41a00 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 1 WAL segments...
I20250113 08:55:00.448628 8863 tablet_copy_client.cc:806] T 44c2ecb48ca14049a051ee94f7d58cd6 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 0 data blocks...
I20250113 08:55:00.448952 8863 tablet_copy_client.cc:670] T 44c2ecb48ca14049a051ee94f7d58cd6 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 1 WAL segments...
I20250113 08:55:00.453038 8873 ts_tablet_manager.cc:927] T c35e4ecbe2cc424089eaa5e0249df483 P 7eb1620e70d84c44b17b3e0aad7c38bc: Initiating tablet copy from peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:55:00.453828 8861 tablet_copy_client.cc:806] T c1e64a047ae649ac9afc1b77608c067f P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 0 data blocks...
I20250113 08:55:00.454155 8861 tablet_copy_client.cc:670] T c1e64a047ae649ac9afc1b77608c067f P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 1 WAL segments...
I20250113 08:55:00.461506 8869 tablet_copy_client.cc:538] T 34cc82c767a44ffab6786f89b8d41a00 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:55:00.467355 8869 tablet_bootstrap.cc:492] T 34cc82c767a44ffab6786f89b8d41a00 P 7eb1620e70d84c44b17b3e0aad7c38bc: Bootstrap starting.
I20250113 08:55:00.469347 8872 tablet_copy_client.cc:323] T 993904453c594e2db077d13f62d9b27a P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.123.2:45341
I20250113 08:55:00.449231 8862 tablet_copy_client.cc:806] T fe2a45627bfe4db0b9cdccfab81e2805 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 0 data blocks...
I20250113 08:55:00.477488 8862 tablet_copy_client.cc:670] T fe2a45627bfe4db0b9cdccfab81e2805 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 1 WAL segments...
I20250113 08:55:00.486279 7991 tablet_copy_service.cc:140] P 1abc86bb580b4d14876623243c9d6594: Received BeginTabletCopySession request for tablet 993904453c594e2db077d13f62d9b27a from peer 7eb1620e70d84c44b17b3e0aad7c38bc ({username='slave'} at 127.7.123.4:49487)
I20250113 08:55:00.486544 7991 tablet_copy_service.cc:161] P 1abc86bb580b4d14876623243c9d6594: Beginning new tablet copy session on tablet 993904453c594e2db077d13f62d9b27a from peer 7eb1620e70d84c44b17b3e0aad7c38bc at {username='slave'} at 127.7.123.4:49487: session id = 7eb1620e70d84c44b17b3e0aad7c38bc-993904453c594e2db077d13f62d9b27a
I20250113 08:55:00.491014 8873 tablet_copy_client.cc:323] T c35e4ecbe2cc424089eaa5e0249df483 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Beginning tablet copy session from remote peer at address 127.7.123.2:45341
I20250113 08:55:00.495458 7990 tablet_copy_service.cc:140] P 1abc86bb580b4d14876623243c9d6594: Received BeginTabletCopySession request for tablet c35e4ecbe2cc424089eaa5e0249df483 from peer 7eb1620e70d84c44b17b3e0aad7c38bc ({username='slave'} at 127.7.123.4:49487)
I20250113 08:55:00.495676 7990 tablet_copy_service.cc:161] P 1abc86bb580b4d14876623243c9d6594: Beginning new tablet copy session on tablet c35e4ecbe2cc424089eaa5e0249df483 from peer 7eb1620e70d84c44b17b3e0aad7c38bc at {username='slave'} at 127.7.123.4:49487: session id = 7eb1620e70d84c44b17b3e0aad7c38bc-c35e4ecbe2cc424089eaa5e0249df483
I20250113 08:55:00.499017 7990 tablet_copy_source_session.cc:215] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:55:00.501940 8873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35e4ecbe2cc424089eaa5e0249df483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:55:00.503393 7991 tablet_copy_source_session.cc:215] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594: Tablet Copy: opened 0 blocks and 1 log segments
I20250113 08:55:00.504892 8872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993904453c594e2db077d13f62d9b27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:55:00.515945 8861 tablet_copy_client.cc:538] T c1e64a047ae649ac9afc1b77608c067f P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:55:00.522091 8861 tablet_bootstrap.cc:492] T c1e64a047ae649ac9afc1b77608c067f P 7eb1620e70d84c44b17b3e0aad7c38bc: Bootstrap starting.
I20250113 08:55:00.510035 8873 tablet_copy_client.cc:806] T c35e4ecbe2cc424089eaa5e0249df483 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 0 data blocks...
I20250113 08:55:00.524536 8873 tablet_copy_client.cc:670] T c35e4ecbe2cc424089eaa5e0249df483 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 1 WAL segments...
I20250113 08:55:00.528312 8863 tablet_copy_client.cc:538] T 44c2ecb48ca14049a051ee94f7d58cd6 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:55:00.532990 8863 tablet_bootstrap.cc:492] T 44c2ecb48ca14049a051ee94f7d58cd6 P 7eb1620e70d84c44b17b3e0aad7c38bc: Bootstrap starting.
I20250113 08:55:00.537108 8862 tablet_copy_client.cc:538] T fe2a45627bfe4db0b9cdccfab81e2805 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:55:00.541579 8862 tablet_bootstrap.cc:492] T fe2a45627bfe4db0b9cdccfab81e2805 P 7eb1620e70d84c44b17b3e0aad7c38bc: Bootstrap starting.
I20250113 08:55:00.551043 8872 tablet_copy_client.cc:806] T 993904453c594e2db077d13f62d9b27a P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 0 data blocks...
I20250113 08:55:00.551503 8872 tablet_copy_client.cc:670] T 993904453c594e2db077d13f62d9b27a P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Starting download of 1 WAL segments...
I20250113 08:55:00.560063 8873 tablet_copy_client.cc:538] T c35e4ecbe2cc424089eaa5e0249df483 P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:55:00.575424 8873 tablet_bootstrap.cc:492] T c35e4ecbe2cc424089eaa5e0249df483 P 7eb1620e70d84c44b17b3e0aad7c38bc: Bootstrap starting.
I20250113 08:55:00.580971 8872 tablet_copy_client.cc:538] T 993904453c594e2db077d13f62d9b27a P 7eb1620e70d84c44b17b3e0aad7c38bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250113 08:55:00.585330 8872 tablet_bootstrap.cc:492] T 993904453c594e2db077d13f62d9b27a P 7eb1620e70d84c44b17b3e0aad7c38bc: Bootstrap starting.
W20250113 08:55:00.654825 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:00.661664 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250113 08:55:00.669430 8406 heartbeater.cc:510] Master 127.7.123.62:33371 requested a full tablet report, sending...
W20250113 08:55:00.675736 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:00.684091 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:00.777333 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:00.788689 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:00.882639 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:00.928959 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:00.961778 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:00.968964 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:00.985958 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:01.057194 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:01.127661 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.150612 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.159590 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.184939 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.310964 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.320493 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.344838 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.405344 7906 consensus_peers.cc:487] T c35e4ecbe2cc424089eaa5e0249df483 P 1abc86bb580b4d14876623243c9d6594 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.495911 7906 consensus_peers.cc:487] T 993904453c594e2db077d13f62d9b27a P 1abc86bb580b4d14876623243c9d6594 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.499692 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.530671 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.536900 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:01.606918 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:55:01.646072 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250113 08:55:01.675206 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 7889
I20250113 08:55:01.676522 8011 generic_service.cc:196] Checking for leaks (request via RPC)
W20250113 08:55:01.691323 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:55:01.703657 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:55:01.855293 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
I20250113 08:55:01.921293 8830 raft_consensus.cc:491] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1abc86bb580b4d14876623243c9d6594)
I20250113 08:55:01.921568 8830 raft_consensus.cc:513] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:01.922483 8830 leader_election.cc:290] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
W20250113 08:55:01.924438 8037 leader_election.cc:336] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
I20250113 08:55:01.934222 8830 raft_consensus.cc:491] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1abc86bb580b4d14876623243c9d6594)
I20250113 08:55:01.934513 8830 raft_consensus.cc:513] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:01.935528 8830 leader_election.cc:290] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
W20250113 08:55:01.937538 8037 leader_election.cc:336] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111)
W20250113 08:55:01.953164 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
I20250113 08:55:01.992096 8862 log.cc:826] T fe2a45627bfe4db0b9cdccfab81e2805 P 7eb1620e70d84c44b17b3e0aad7c38bc: Log is configured to *not* fsync() on all Append() calls
W20250113 08:55:02.028352 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
W20250113 08:55:02.031106 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
W20250113 08:55:02.088979 8037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111) [suppressed 194 similar messages]
W20250113 08:55:02.099460 8037 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250113 08:55:02.123724 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250113 08:55:02.194429 8830 consensus_queue.cc:579] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.064s)
I20250113 08:55:02.197965 8890 consensus_queue.cc:579] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.067s)
W20250113 08:55:02.199251 8037 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250113 08:55:02.224155 8890 consensus_queue.cc:579] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.094s)
I20250113 08:55:02.229507 8886 consensus_queue.cc:579] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 [LEADER]: Leader has been unable to successfully communicate with peer aea19a9cd21144508fdcfd7b3570a0e1 for more than 2 seconds (2.092s)
W20250113 08:55:02.236459 8037 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439): Couldn't send request to peer aea19a9cd21144508fdcfd7b3570a0e1. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:46439: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250113 08:55:02.271396 7907 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.001s sys 0.000s
W20250113 08:55:02.271487 7907 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.553s user 0.001s sys 0.000s
I20250113 08:55:02.271322 8849 raft_consensus.cc:491] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 81f9c074161846b1b2a552cb12d3ef30)
I20250113 08:55:02.272547 8849 raft_consensus.cc:513] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2073 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:02.273339 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 7889
I20250113 08:55:02.273355 8849 leader_election.cc:290] T 34cc82c767a44ffab6786f89b8d41a00 P 1abc86bb580b4d14876623243c9d6594 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 81f9c074161846b1b2a552cb12d3ef30 (127.7.123.3:46625)
W20250113 08:55:02.366912 8037 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 7eb1620e70d84c44b17b3e0aad7c38bc (127.7.123.4:38355): Couldn't send request to peer 7eb1620e70d84c44b17b3e0aad7c38bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
W20250113 08:55:02.367251 8038 connection.cc:537] client connection to 127.7.123.2:45341 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:55:02.367358 8038 leader_election.cc:336] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:55:02.367400 8481 connection.cc:537] server connection from 127.7.123.2:57025 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:55:02.367455 8038 leader_election.cc:304] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81f9c074161846b1b2a552cb12d3ef30; no voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1
W20250113 08:55:02.367851 8038 leader_election.cc:336] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:55:02.367993 8038 leader_election.cc:304] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 81f9c074161846b1b2a552cb12d3ef30; no voters: 1abc86bb580b4d14876623243c9d6594, aea19a9cd21144508fdcfd7b3570a0e1
I20250113 08:55:02.368148 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 8020
I20250113 08:55:02.368607 8830 raft_consensus.cc:2743] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:55:02.368824 8830 raft_consensus.cc:2743] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:55:02.369366 8142 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:55:02.955928 8886 raft_consensus.cc:491] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1abc86bb580b4d14876623243c9d6594)
I20250113 08:55:02.956249 8886 raft_consensus.cc:513] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:02.957096 8886 leader_election.cc:290] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
I20250113 08:55:02.960119 8830 raft_consensus.cc:491] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1abc86bb580b4d14876623243c9d6594)
I20250113 08:55:02.960489 8830 raft_consensus.cc:513] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2077 OBSOLETE_local: false peers { permanent_uuid: "aea19a9cd21144508fdcfd7b3570a0e1" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 46439 } } peers { permanent_uuid: "1abc86bb580b4d14876623243c9d6594" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 45341 } } peers { permanent_uuid: "81f9c074161846b1b2a552cb12d3ef30" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 46625 } } peers { permanent_uuid: "7eb1620e70d84c44b17b3e0aad7c38bc" member_type: NON_VOTER last_known_addr { host: "127.7.123.4" port: 38355 } attrs { promote: true } }
I20250113 08:55:02.961740 8830 leader_election.cc:290] T 993904453c594e2db077d13f62d9b27a P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aea19a9cd21144508fdcfd7b3570a0e1 (127.7.123.1:46439), 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341)
W20250113 08:55:02.965509 8038 consensus_peers.cc:487] T fe2a45627bfe4db0b9cdccfab81e2805 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Couldn't send request to peer 1abc86bb580b4d14876623243c9d6594. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.2:45341: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:02.965803 8038 consensus_peers.cc:487] T 44c2ecb48ca14049a051ee94f7d58cd6 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Couldn't send request to peer 1abc86bb580b4d14876623243c9d6594. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.2:45341: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:02.966064 8038 consensus_peers.cc:487] T c1e64a047ae649ac9afc1b77608c067f P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Couldn't send request to peer 1abc86bb580b4d14876623243c9d6594. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.2:45341: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:02.966264 8038 consensus_peers.cc:487] T 34cc82c767a44ffab6786f89b8d41a00 P 81f9c074161846b1b2a552cb12d3ef30 -> Peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Couldn't send request to peer 1abc86bb580b4d14876623243c9d6594. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.2:45341: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:02.967882 8038 leader_election.cc:336] T c35e4ecbe2cc424089eaa5e0249df483 P 81f9c074161846b1b2a552cb12d3ef30 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1abc86bb580b4d14876623243c9d6594 (127.7.123.2:45341): Network error: Client connection negotiation failed: client connection to 127.7.123.2:45341: connect: Connection refused (error 111)
I20250113 08:55:02.969187 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 8020
I20250113 08:55:03.067129 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 8238
W20250113 08:55:03.067653 8290 connection.cc:537] server connection from 127.7.123.3:57587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:55:03.070725 8401 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:55:03.298997 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 8238
I20250113 08:55:03.332319 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 8462
I20250113 08:55:03.333823 8527 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:55:03.452330 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 8462
2025-01-13T08:55:03Z chronyd exiting
[ OK ] MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate (54457 ms)
[----------] 1 test from MaintenanceModeRF3ITest (54457 ms total)
[----------] 1 test from RollingRestartArgs/RollingRestartITest
[ RUN ] RollingRestartArgs/RollingRestartITest.TestWorkloads/4
2025-01-13T08:55:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-13T08:55:03Z Disabled control of system clock
I20250113 08:55:03.575155 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.123.62:34471
--webserver_interface=127.7.123.62
--webserver_port=0
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.123.62:34471
--location_mapping_cmd=/tmp/dist-test-taskmZWr4w/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/location-assignment.state --map /L0:4
--master_client_location_assignment_enabled=false with env {}
W20250113 08:55:03.982748 8908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:03.983280 8908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:03.983392 8908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:03.992122 8908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250113 08:55:03.992281 8908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:03.992332 8908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250113 08:55:03.992422 8908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 08:55:04.003358 8908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/wal
--location_mapping_cmd=/tmp/dist-test-taskmZWr4w/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/location-assignment.state --map /L0:4
--ipki_ca_key_size=768
--master_addresses=127.7.123.62:34471
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.123.62:34471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.123.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:04.005421 8908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:04.007712 8908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:04.021451 8913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:04.022042 8916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:04.022256 8914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:04.023345 8908 server_base.cc:1034] running on GCE node
I20250113 08:55:04.024466 8908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:04.026516 8908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:04.027822 8908 hybrid_clock.cc:648] HybridClock initialized: now 1736758504027709 us; error 90 us; skew 500 ppm
I20250113 08:55:04.028291 8908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:04.036432 8908 webserver.cc:458] Webserver started at http://127.7.123.62:44425/ using document root <none> and password file <none>
I20250113 08:55:04.037495 8908 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:04.037645 8908 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:04.038046 8908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:55:04.041891 8908 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/data/instance:
uuid: "611dd4445ed147848d00225ecc800448"
format_stamp: "Formatted at 2025-01-13 08:55:04 on dist-test-slave-vxbw"
I20250113 08:55:04.043171 8908 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/wal/instance:
uuid: "611dd4445ed147848d00225ecc800448"
format_stamp: "Formatted at 2025-01-13 08:55:04 on dist-test-slave-vxbw"
I20250113 08:55:04.050118 8908 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250113 08:55:04.054428 8922 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:04.056205 8908 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250113 08:55:04.056432 8908 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/wal
uuid: "611dd4445ed147848d00225ecc800448"
format_stamp: "Formatted at 2025-01-13 08:55:04 on dist-test-slave-vxbw"
I20250113 08:55:04.056701 8908 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:04.106896 8908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:04.108016 8908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:04.108379 8908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:04.130003 8908 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.62:34471
I20250113 08:55:04.130024 8973 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.62:34471 every 8 connection(s)
I20250113 08:55:04.131920 8908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/data/info.pb
I20250113 08:55:04.132289 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 8908
I20250113 08:55:04.132514 7660 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/master-0/wal/instance
I20250113 08:55:04.137367 8974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:55:04.151701 8974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448: Bootstrap starting.
I20250113 08:55:04.155948 8974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448: Neither blocks nor log segments found. Creating new log.
I20250113 08:55:04.157169 8974 log.cc:826] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448: Log is configured to *not* fsync() on all Append() calls
I20250113 08:55:04.161082 8974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448: No bootstrap required, opened a new log
I20250113 08:55:04.168592 8974 raft_consensus.cc:357] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611dd4445ed147848d00225ecc800448" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 34471 } }
I20250113 08:55:04.169044 8974 raft_consensus.cc:383] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:55:04.169142 8974 raft_consensus.cc:738] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 611dd4445ed147848d00225ecc800448, State: Initialized, Role: FOLLOWER
I20250113 08:55:04.169790 8974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611dd4445ed147848d00225ecc800448" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 34471 } }
I20250113 08:55:04.170013 8974 raft_consensus.cc:397] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 08:55:04.170125 8974 raft_consensus.cc:491] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 08:55:04.170272 8974 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:55:04.172533 8974 raft_consensus.cc:513] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611dd4445ed147848d00225ecc800448" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 34471 } }
I20250113 08:55:04.173236 8974 leader_election.cc:304] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 611dd4445ed147848d00225ecc800448; no voters:
I20250113 08:55:04.173935 8974 leader_election.cc:290] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 08:55:04.174171 8979 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:55:04.174916 8979 raft_consensus.cc:695] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [term 1 LEADER]: Becoming Leader. State: Replica: 611dd4445ed147848d00225ecc800448, State: Running, Role: LEADER
I20250113 08:55:04.175580 8979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611dd4445ed147848d00225ecc800448" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 34471 } }
I20250113 08:55:04.176666 8974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 08:55:04.179232 8980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "611dd4445ed147848d00225ecc800448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611dd4445ed147848d00225ecc800448" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 34471 } } }
I20250113 08:55:04.179729 8980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [sys.catalog]: This master's current role is: LEADER
I20250113 08:55:04.180385 8981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 611dd4445ed147848d00225ecc800448. Latest consensus state: current_term: 1 leader_uuid: "611dd4445ed147848d00225ecc800448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611dd4445ed147848d00225ecc800448" member_type: VOTER last_known_addr { host: "127.7.123.62" port: 34471 } } }
I20250113 08:55:04.180662 8981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448 [sys.catalog]: This master's current role is: LEADER
I20250113 08:55:04.181784 8987 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 08:55:04.189191 8987 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 08:55:04.201247 8987 catalog_manager.cc:1348] Generated new cluster ID: c44bf7a45dab4538b53df424411ee303
I20250113 08:55:04.201421 8987 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 08:55:04.216972 8987 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 08:55:04.218551 8987 catalog_manager.cc:1505] Loading token signing keys...
I20250113 08:55:04.232559 8987 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 611dd4445ed147848d00225ecc800448: Generated new TSK 0
I20250113 08:55:04.233754 8987 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 08:55:04.247422 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.1:0
--local_ip_for_outbound_sockets=127.7.123.1
--webserver_interface=127.7.123.1
--webserver_port=0
--tserver_master_addrs=127.7.123.62:34471
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:55:04.690683 8998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:04.691118 8998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:04.691293 8998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:04.700040 8998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:04.700343 8998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.1
I20250113 08:55:04.710718 8998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.123.1
--webserver_port=0
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.1
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:04.712759 8998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:04.715081 8998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:04.729691 9003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:04.729879 9006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:04.730664 9004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:04.731684 8998 server_base.cc:1034] running on GCE node
I20250113 08:55:04.732463 8998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:04.733788 8998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:04.735211 8998 hybrid_clock.cc:648] HybridClock initialized: now 1736758504735110 us; error 71 us; skew 500 ppm
I20250113 08:55:04.735625 8998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:04.738584 8998 webserver.cc:458] Webserver started at http://127.7.123.1:33665/ using document root <none> and password file <none>
I20250113 08:55:04.739585 8998 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:04.739754 8998 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:04.740134 8998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:55:04.743458 8998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/instance:
uuid: "e3f389d6e3484a13af0780989a117f02"
format_stamp: "Formatted at 2025-01-13 08:55:04 on dist-test-slave-vxbw"
I20250113 08:55:04.744339 8998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal/instance:
uuid: "e3f389d6e3484a13af0780989a117f02"
format_stamp: "Formatted at 2025-01-13 08:55:04 on dist-test-slave-vxbw"
I20250113 08:55:04.751086 8998 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250113 08:55:04.755532 9012 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:04.757544 8998 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250113 08:55:04.757854 8998 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
uuid: "e3f389d6e3484a13af0780989a117f02"
format_stamp: "Formatted at 2025-01-13 08:55:04 on dist-test-slave-vxbw"
I20250113 08:55:04.758170 8998 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:04.776824 8998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:04.777846 8998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:04.778201 8998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:04.779366 8998 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:04.781345 8998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:55:04.781437 8998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:04.781560 8998 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:55:04.781617 8998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:04.819154 8998 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.1:38389
I20250113 08:55:04.819190 9124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.1:38389 every 8 connection(s)
I20250113 08:55:04.820837 8998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
I20250113 08:55:04.822314 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 8998
I20250113 08:55:04.822531 7660 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal/instance
I20250113 08:55:04.826390 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.2:0
--local_ip_for_outbound_sockets=127.7.123.2
--webserver_interface=127.7.123.2
--webserver_port=0
--tserver_master_addrs=127.7.123.62:34471
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250113 08:55:04.838673 9125 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:04.839110 9125 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:04.840189 9125 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:04.912753 8939 ts_manager.cc:194] Registered new tserver with Master: e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389)
I20250113 08:55:04.914783 8939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.1:58563
W20250113 08:55:05.257001 9129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:05.257381 9129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:05.257520 9129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:05.266259 9129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:05.266551 9129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.2
I20250113 08:55:05.277120 9129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.123.2
--webserver_port=0
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.2
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:05.279182 9129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:05.281342 9129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:05.295879 9136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:05.296646 9135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:05.297228 9138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:05.297669 9129 server_base.cc:1034] running on GCE node
I20250113 08:55:05.298429 9129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:05.299749 9129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:05.301055 9129 hybrid_clock.cc:648] HybridClock initialized: now 1736758505300966 us; error 74 us; skew 500 ppm
I20250113 08:55:05.301447 9129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:05.304368 9129 webserver.cc:458] Webserver started at http://127.7.123.2:39675/ using document root <none> and password file <none>
I20250113 08:55:05.305343 9129 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:05.305497 9129 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:05.305843 9129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:55:05.309095 9129 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/instance:
uuid: "5f8ea89390544489a231c733997d2d9a"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:55:05.310097 9129 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal/instance:
uuid: "5f8ea89390544489a231c733997d2d9a"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:55:05.316988 9129 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250113 08:55:05.321514 9144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:05.323482 9129 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250113 08:55:05.323747 9129 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
uuid: "5f8ea89390544489a231c733997d2d9a"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:55:05.324023 9129 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:05.340533 9129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:05.341579 9129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:05.341949 9129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:05.343256 9129 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:05.345227 9129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:55:05.345350 9129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:05.345492 9129 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:55:05.345551 9129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:05.381611 9129 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.2:43333
I20250113 08:55:05.381644 9256 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.2:43333 every 8 connection(s)
I20250113 08:55:05.383322 9129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
I20250113 08:55:05.384146 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 9129
I20250113 08:55:05.384369 7660 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal/instance
I20250113 08:55:05.388679 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.3:0
--local_ip_for_outbound_sockets=127.7.123.3
--webserver_interface=127.7.123.3
--webserver_port=0
--tserver_master_addrs=127.7.123.62:34471
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250113 08:55:05.402642 9257 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:05.403087 9257 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:05.404088 9257 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:05.473667 8939 ts_manager.cc:194] Registered new tserver with Master: 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333)
I20250113 08:55:05.474709 8939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.2:57749
W20250113 08:55:05.827410 9261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:05.827832 9261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:05.827994 9261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:05.836679 9261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:05.836966 9261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.3
I20250113 08:55:05.847558 9261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.123.3
--webserver_port=0
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.3
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:05.849659 9261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:05.852115 9261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:05.867587 9267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:05.868206 9268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:05.868712 9270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:05.869052 9261 server_base.cc:1034] running on GCE node
I20250113 08:55:05.869920 9261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:05.871204 9261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:05.872476 9261 hybrid_clock.cc:648] HybridClock initialized: now 1736758505872395 us; error 54 us; skew 500 ppm
I20250113 08:55:05.872888 9261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:05.875749 9261 webserver.cc:458] Webserver started at http://127.7.123.3:35523/ using document root <none> and password file <none>
I20250113 08:55:05.876745 9261 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:05.876896 9261 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:05.877308 9261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:55:05.880843 9261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/instance:
uuid: "cf369533d8ad453f9d6c47cad3706dab"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:55:05.881884 9261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal/instance:
uuid: "cf369533d8ad453f9d6c47cad3706dab"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:55:05.888746 9261 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250113 08:55:05.893239 9276 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:05.895123 9261 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250113 08:55:05.895375 9261 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
uuid: "cf369533d8ad453f9d6c47cad3706dab"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:55:05.895615 9261 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:05.913326 9261 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:05.914390 9261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:05.914734 9261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:05.915868 9261 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:05.917877 9261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:55:05.917991 9261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:05.918145 9261 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:55:05.918205 9261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:05.918500 9125 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:55:05.955034 9261 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.3:43245
I20250113 08:55:05.955078 9388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.3:43245 every 8 connection(s)
I20250113 08:55:05.956710 9261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
I20250113 08:55:05.963061 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 9261
I20250113 08:55:05.963266 7660 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal/instance
I20250113 08:55:05.967370 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.4:0
--local_ip_for_outbound_sockets=127.7.123.4
--webserver_interface=127.7.123.4
--webserver_port=0
--tserver_master_addrs=127.7.123.62:34471
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250113 08:55:05.974678 9389 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:05.975155 9389 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:05.976353 9389 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:06.029507 8939 ts_manager.cc:194] Registered new tserver with Master: cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:06.030668 8939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.3:51673
W20250113 08:55:06.394034 9393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:06.394414 9393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:06.394558 9393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:06.403232 9393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:06.403540 9393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.4
I20250113 08:55:06.414043 9393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.7.123.4
--webserver_port=0
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.4
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:06.416033 9393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:06.418172 9393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:06.432111 9402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:06.432849 9399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:06.433468 9400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:06.434587 9393 server_base.cc:1034] running on GCE node
I20250113 08:55:06.435351 9393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:06.436501 9393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:06.437741 9393 hybrid_clock.cc:648] HybridClock initialized: now 1736758506437657 us; error 60 us; skew 500 ppm
I20250113 08:55:06.438154 9393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:06.441334 9393 webserver.cc:458] Webserver started at http://127.7.123.4:45007/ using document root <none> and password file <none>
I20250113 08:55:06.442364 9393 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:06.442529 9393 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:06.442981 9393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:55:06.446367 9393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/instance:
uuid: "00524bc1b1a34b61af38c9f2bded4031"
format_stamp: "Formatted at 2025-01-13 08:55:06 on dist-test-slave-vxbw"
I20250113 08:55:06.447450 9393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal/instance:
uuid: "00524bc1b1a34b61af38c9f2bded4031"
format_stamp: "Formatted at 2025-01-13 08:55:06 on dist-test-slave-vxbw"
I20250113 08:55:06.454483 9393 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250113 08:55:06.459314 9408 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:06.461238 9393 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250113 08:55:06.461489 9393 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
uuid: "00524bc1b1a34b61af38c9f2bded4031"
format_stamp: "Formatted at 2025-01-13 08:55:06 on dist-test-slave-vxbw"
I20250113 08:55:06.461740 9393 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:06.478974 9257 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:55:06.502157 9393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:06.503170 9393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:06.503499 9393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:06.504594 9393 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:06.506404 9393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:55:06.506507 9393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:06.506644 9393 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:55:06.506702 9393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:06.544153 9393 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.4:34591
I20250113 08:55:06.544190 9520 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.4:34591 every 8 connection(s)
I20250113 08:55:06.546010 9393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
I20250113 08:55:06.551347 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 9393
I20250113 08:55:06.551565 7660 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal/instance
I20250113 08:55:06.562373 9521 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:06.562786 9521 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:06.564002 9521 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:06.621279 8939 ts_manager.cc:194] Registered new tserver with Master: 00524bc1b1a34b61af38c9f2bded4031 (127.7.123.4:34591)
I20250113 08:55:06.622390 8939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.4:47557
I20250113 08:55:06.632048 7660 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250113 08:55:06.654371 7660 test_util.cc:274] Using random seed: -1713796489
I20250113 08:55:06.680141 8939 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39862:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 08:55:06.718482 9060 tablet_service.cc:1467] Processing CreateTablet for tablet b13658abfdf04a8c9fcc1f08c15ff92a (DEFAULT_TABLE table=test-workload [id=314c4c0b84f54d1698be42a4edc0fef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 08:55:06.718777 9192 tablet_service.cc:1467] Processing CreateTablet for tablet b13658abfdf04a8c9fcc1f08c15ff92a (DEFAULT_TABLE table=test-workload [id=314c4c0b84f54d1698be42a4edc0fef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 08:55:06.720324 9192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13658abfdf04a8c9fcc1f08c15ff92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:55:06.720368 9060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13658abfdf04a8c9fcc1f08c15ff92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:55:06.719897 9324 tablet_service.cc:1467] Processing CreateTablet for tablet b13658abfdf04a8c9fcc1f08c15ff92a (DEFAULT_TABLE table=test-workload [id=314c4c0b84f54d1698be42a4edc0fef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 08:55:06.721580 9324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13658abfdf04a8c9fcc1f08c15ff92a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:55:06.735312 9546 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap starting.
I20250113 08:55:06.738451 9547 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap starting.
I20250113 08:55:06.740635 9546 tablet_bootstrap.cc:654] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Neither blocks nor log segments found. Creating new log.
I20250113 08:55:06.741175 9548 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap starting.
I20250113 08:55:06.743261 9547 tablet_bootstrap.cc:654] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Neither blocks nor log segments found. Creating new log.
I20250113 08:55:06.743916 9546 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Log is configured to *not* fsync() on all Append() calls
I20250113 08:55:06.744935 9547 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Log is configured to *not* fsync() on all Append() calls
I20250113 08:55:06.745465 9548 tablet_bootstrap.cc:654] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Neither blocks nor log segments found. Creating new log.
I20250113 08:55:06.746889 9548 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Log is configured to *not* fsync() on all Append() calls
I20250113 08:55:06.747992 9546 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: No bootstrap required, opened a new log
I20250113 08:55:06.748349 9547 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: No bootstrap required, opened a new log
I20250113 08:55:06.748412 9546 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s
I20250113 08:55:06.748679 9547 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20250113 08:55:06.749966 9548 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: No bootstrap required, opened a new log
I20250113 08:55:06.750290 9548 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20250113 08:55:06.759156 9548 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:06.759387 9547 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:06.759387 9546 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:06.759865 9548 raft_consensus.cc:383] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:55:06.759865 9547 raft_consensus.cc:383] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:55:06.759865 9546 raft_consensus.cc:383] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:55:06.759995 9548 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf369533d8ad453f9d6c47cad3706dab, State: Initialized, Role: FOLLOWER
I20250113 08:55:06.760008 9547 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f389d6e3484a13af0780989a117f02, State: Initialized, Role: FOLLOWER
I20250113 08:55:06.759996 9546 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8ea89390544489a231c733997d2d9a, State: Initialized, Role: FOLLOWER
I20250113 08:55:06.761029 9548 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:06.761044 9547 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:06.761911 9546 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:06.765729 9547 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
I20250113 08:55:06.767136 9389 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:55:06.770530 9548 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Time spent starting tablet: real 0.020s user 0.013s sys 0.005s
I20250113 08:55:06.772907 9546 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Time spent starting tablet: real 0.024s user 0.016s sys 0.005s
I20250113 08:55:06.774401 9553 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:55:06.774758 9553 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:06.778492 9553 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:06.787271 9212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a" is_pre_election: true
I20250113 08:55:06.788065 9212 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f389d6e3484a13af0780989a117f02 in term 0.
I20250113 08:55:06.788969 9015 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a, e3f389d6e3484a13af0780989a117f02; no voters:
I20250113 08:55:06.788998 9344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
I20250113 08:55:06.789511 9553 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:55:06.789609 9344 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f389d6e3484a13af0780989a117f02 in term 0.
I20250113 08:55:06.789661 9553 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:55:06.789763 9553 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:55:06.792176 9553 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:06.792925 9553 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:06.793491 9344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab"
I20250113 08:55:06.793495 9212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a"
I20250113 08:55:06.793709 9344 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:55:06.793718 9212 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:55:06.796262 9344 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f389d6e3484a13af0780989a117f02 in term 1.
I20250113 08:55:06.796432 9212 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f389d6e3484a13af0780989a117f02 in term 1.
I20250113 08:55:06.796926 9016 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf369533d8ad453f9d6c47cad3706dab, e3f389d6e3484a13af0780989a117f02; no voters:
I20250113 08:55:06.797299 9553 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:55:06.797842 9553 raft_consensus.cc:695] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 1 LEADER]: Becoming Leader. State: Replica: e3f389d6e3484a13af0780989a117f02, State: Running, Role: LEADER
I20250113 08:55:06.798538 9553 consensus_queue.cc:237] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:06.806299 8939 catalog_manager.cc:5526] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3f389d6e3484a13af0780989a117f02 (127.7.123.1). New cstate: current_term: 1 leader_uuid: "e3f389d6e3484a13af0780989a117f02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } health_report { overall_health: UNKNOWN } } }
W20250113 08:55:06.824605 9126 tablet.cc:2367] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:55:06.834122 7660 maintenance_mode-itest.cc:745] Restarting batch of 4 tservers: 5f8ea89390544489a231c733997d2d9a,e3f389d6e3484a13af0780989a117f02,00524bc1b1a34b61af38c9f2bded4031,cf369533d8ad453f9d6c47cad3706dab
W20250113 08:55:06.886641 9258 tablet.cc:2367] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:55:06.898705 9553 consensus_queue.cc:1035] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:55:06.906524 9553 consensus_queue.cc:1035] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:55:07.625823 9521 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:55:07.979229 8930 ts_manager.cc:292] Set tserver state for e3f389d6e3484a13af0780989a117f02 to MAINTENANCE_MODE
I20250113 08:55:08.228998 8931 ts_manager.cc:292] Set tserver state for 00524bc1b1a34b61af38c9f2bded4031 to MAINTENANCE_MODE
I20250113 08:55:09.336392 8931 ts_manager.cc:292] Set tserver state for 5f8ea89390544489a231c733997d2d9a to MAINTENANCE_MODE
I20250113 08:55:09.351099 8937 ts_manager.cc:292] Set tserver state for cf369533d8ad453f9d6c47cad3706dab to MAINTENANCE_MODE
I20250113 08:55:10.596730 9192 tablet_service.cc:1422] Tablet server 5f8ea89390544489a231c733997d2d9a set to quiescing
I20250113 08:55:10.597043 9192 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:11.304409 9060 tablet_service.cc:1422] Tablet server e3f389d6e3484a13af0780989a117f02 set to quiescing
I20250113 08:55:11.304734 9060 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:55:11.319940 9553 raft_consensus.cc:988] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: : Instructing follower cf369533d8ad453f9d6c47cad3706dab to start an election
I20250113 08:55:11.320266 9557 raft_consensus.cc:1076] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 1 LEADER]: Signalling peer cf369533d8ad453f9d6c47cad3706dab to start an election
I20250113 08:55:11.321439 9344 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a"
dest_uuid: "cf369533d8ad453f9d6c47cad3706dab"
from {username='slave'} at 127.7.123.1:34629
I20250113 08:55:11.321991 9344 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:55:11.322180 9344 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:55:11.334712 9344 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:11.349118 9344 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 2 election: Requested vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333)
I20250113 08:55:11.391710 9344 raft_consensus.cc:1235] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 2 FOLLOWER]: Rejecting Update request from peer e3f389d6e3484a13af0780989a117f02 for earlier term 1. Current term is 2. Ops: [1.147-1.148]
I20250113 08:55:11.392904 9553 consensus_queue.cc:1046] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 }, Status: INVALID_TERM, Last received: 1.146, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I20250113 08:55:11.393342 9553 raft_consensus.cc:3049] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:55:11.393487 9553 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e3f389d6e3484a13af0780989a117f02, State: Running, Role: LEADER
I20250113 08:55:11.393952 9553 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 1.151, Last appended by leader: 151, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:11.394644 9553 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 1 FOLLOWER]: Advancing to term 2
W20250113 08:55:11.405647 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:11.430895 9210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "cf369533d8ad453f9d6c47cad3706dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 146 } } ignore_live_leader: true dest_uuid: "5f8ea89390544489a231c733997d2d9a"
I20250113 08:55:11.431216 9210 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:55:11.436290 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "cf369533d8ad453f9d6c47cad3706dab" candidate_term: 2 candidate_status { last_received { term: 1 index: 146 } } ignore_live_leader: true dest_uuid: "e3f389d6e3484a13af0780989a117f02"
I20250113 08:55:11.436913 9080 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cf369533d8ad453f9d6c47cad3706dab for term 2 because replica has last-logged OpId of term: 1 index: 151, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 146.
I20250113 08:55:11.437742 9210 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cf369533d8ad453f9d6c47cad3706dab for term 2 because replica has last-logged OpId of term: 1 index: 151, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 146.
I20250113 08:55:11.438670 9279 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf369533d8ad453f9d6c47cad3706dab; no voters: 5f8ea89390544489a231c733997d2d9a, e3f389d6e3484a13af0780989a117f02
I20250113 08:55:11.460532 9697 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250113 08:55:11.462309 9324 tablet_service.cc:1422] Tablet server cf369533d8ad453f9d6c47cad3706dab set to quiescing
I20250113 08:55:11.462572 9324 tablet_service.cc:1429] Tablet server has 0 leaders and 2 scanners
W20250113 08:55:11.489226 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:11.497344 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:11.527710 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:11.536441 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:11.544670 9302 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:11.559633 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:11.572077 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:11.584082 9302 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:11.600498 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:11.617461 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:11.632823 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:11.654510 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:11.672896 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:11.690131 9701 raft_consensus.cc:668] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: failed to trigger leader election: Illegal state: leader elections are disabled
W20250113 08:55:11.693183 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:11.714619 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:11.737738 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:11.755302 9557 raft_consensus.cc:668] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: failed to trigger leader election: Illegal state: leader elections are disabled
W20250113 08:55:11.761835 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:11.786332 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:11.809693 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:11.838291 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:11.869148 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:11.897253 9697 raft_consensus.cc:668] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: failed to trigger leader election: Illegal state: leader elections are disabled
W20250113 08:55:11.897812 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:11.926558 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:11.962342 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:11.999161 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:12.029709 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:12.064242 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:12.077785 9456 tablet_service.cc:1422] Tablet server 00524bc1b1a34b61af38c9f2bded4031 set to quiescing
I20250113 08:55:12.078078 9456 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:12.097569 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:12.131202 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:12.172788 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:12.209880 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:12.249171 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:12.292804 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:12.333031 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:12.376514 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:12.422889 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:12.466943 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:12.512387 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:12.558120 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:12.604038 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:12.651171 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:12.701303 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:12.750039 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:12.803071 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:12.858321 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:12.913472 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:12.966084 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:13.028098 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:13.086438 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:13.142082 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:13.202075 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:13.207105 9060 tablet_service.cc:1422] Tablet server e3f389d6e3484a13af0780989a117f02 set to quiescing
I20250113 08:55:13.207435 9060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:13.261575 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:13.320566 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:13.378563 9324 tablet_service.cc:1422] Tablet server cf369533d8ad453f9d6c47cad3706dab set to quiescing
I20250113 08:55:13.378892 9324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:13.381371 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:13.443441 9172 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41174: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:55:13.468118 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 9129
I20250113 08:55:13.476919 9252 generic_service.cc:196] Checking for leaks (request via RPC)
W20250113 08:55:13.502689 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:13.567689 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:13.583907 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 9129
I20250113 08:55:13.607576 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.2:43333
--local_ip_for_outbound_sockets=127.7.123.2
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=39675
--webserver_interface=127.7.123.2
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:55:13.629418 9533 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333) has failed: Network error: Client connection negotiation failed: client connection to 127.7.123.2:43333: connect: Connection refused (error 111)
W20250113 08:55:13.695086 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:13.762135 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:13.895004 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:13.964471 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:14.029059 9741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:14.029479 9741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:14.029647 9741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250113 08:55:14.037202 9562 meta_cache.cc:1510] marking tablet server 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333) as failed
W20250113 08:55:14.040409 9741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:14.040760 9741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.2
I20250113 08:55:14.051523 9741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.2:43333
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.123.2
--webserver_port=39675
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.2
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:14.053597 9741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:14.056717 9741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:14.073367 9748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:14.074049 9747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:14.074175 9750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:14.075520 9741 server_base.cc:1034] running on GCE node
I20250113 08:55:14.076417 9741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:14.077667 9741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:14.079028 9741 hybrid_clock.cc:648] HybridClock initialized: now 1736758514078920 us; error 71 us; skew 500 ppm
I20250113 08:55:14.079457 9741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:14.082360 9741 webserver.cc:458] Webserver started at http://127.7.123.2:39675/ using document root <none> and password file <none>
I20250113 08:55:14.083442 9741 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:14.083635 9741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:14.090677 9741 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20250113 08:55:14.094861 9756 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:14.097107 9741 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
W20250113 08:55:14.098282 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:14.097373 9741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
uuid: "5f8ea89390544489a231c733997d2d9a"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:55:14.098855 9741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:14.149385 9741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:14.150527 9741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:14.150907 9741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:14.152061 9741 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:14.154489 9763 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:55:14.161334 9741 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:55:14.161522 9741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20250113 08:55:14.161692 9741 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:55:14.165683 9741 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:55:14.165809 9741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20250113 08:55:14.166270 9763 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap starting.
W20250113 08:55:14.171689 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:14.181897 9563 meta_cache.cc:1510] marking tablet server 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333) as failed
I20250113 08:55:14.228474 9741 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.2:43333
I20250113 08:55:14.228516 9869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.2:43333 every 8 connection(s)
I20250113 08:55:14.230525 9741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
I20250113 08:55:14.239531 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 9741
I20250113 08:55:14.239857 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 8998
W20250113 08:55:14.242122 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:14.250303 9120 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:55:14.250463 9870 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:14.251019 9870 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:14.251232 9763 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Log is configured to *not* fsync() on all Append() calls
I20250113 08:55:14.252501 9870 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:14.256629 8937 ts_manager.cc:194] Re-registered known tserver with Master: 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333)
I20250113 08:55:14.258148 8937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.2:34725
W20250113 08:55:14.384343 9040 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49852: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:14.384477 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 8998
I20250113 08:55:14.410008 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.1:38389
--local_ip_for_outbound_sockets=127.7.123.1
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=33665
--webserver_interface=127.7.123.1
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:55:14.448731 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.448731 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.469902 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.471129 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.520218 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.523267 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.555824 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.563089 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.573738 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.621184 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.633759 9304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.639163 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.642249 9534 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389) has failed: Network error: Client connection negotiation failed: client connection to 127.7.123.1:38389: connect: Connection refused (error 111)
W20250113 08:55:14.705965 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.708791 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:14.754647 9763 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=151 overwritten=0 applied=149 ignored=0} inserts{seen=7400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 08:55:14.755491 9763 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap complete.
I20250113 08:55:14.757177 9763 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Time spent bootstrapping tablet: real 0.591s user 0.529s sys 0.056s
I20250113 08:55:14.763202 9763 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 2 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:14.767108 9763 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8ea89390544489a231c733997d2d9a, State: Initialized, Role: FOLLOWER
I20250113 08:55:14.768018 9763 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 149, Last appended: 1.151, Last appended by leader: 151, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:14.769297 9870 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:55:14.770295 9763 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20250113 08:55:14.782315 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.787793 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:14.865114 9785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:14.872375 9878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:14.872738 9878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:14.872884 9878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:14.881616 9878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:14.881929 9878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.1
I20250113 08:55:14.892684 9878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.1:38389
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.123.1
--webserver_port=33665
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.1
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:14.894577 9878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:14.896742 9878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:14.912175 9886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:14.912436 9887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:14.912981 9889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:14.914108 9878 server_base.cc:1034] running on GCE node
I20250113 08:55:14.914906 9878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:14.916155 9878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:14.917479 9878 hybrid_clock.cc:648] HybridClock initialized: now 1736758514917394 us; error 56 us; skew 500 ppm
I20250113 08:55:14.917872 9878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:14.920776 9878 webserver.cc:458] Webserver started at http://127.7.123.1:33665/ using document root <none> and password file <none>
I20250113 08:55:14.921720 9878 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:14.921906 9878 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:14.928474 9878 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250113 08:55:14.932291 9895 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:14.934165 9878 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250113 08:55:14.934494 9878 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
uuid: "e3f389d6e3484a13af0780989a117f02"
format_stamp: "Formatted at 2025-01-13 08:55:04 on dist-test-slave-vxbw"
I20250113 08:55:14.935715 9878 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250113 08:55:14.945987 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:14.957290 9878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:14.958343 9878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:14.958648 9878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:14.959859 9878 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:14.962532 9902 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:55:14.971431 9878 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:55:14.971632 9878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20250113 08:55:14.971796 9878 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:55:14.975545 9878 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:55:14.975692 9878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20250113 08:55:14.976066 9902 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap starting.
I20250113 08:55:15.029532 9878 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.1:38389
I20250113 08:55:15.029546 10008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.1:38389 every 8 connection(s)
I20250113 08:55:15.032155 9878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
I20250113 08:55:15.032629 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 9878
I20250113 08:55:15.032878 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 9393
I20250113 08:55:15.061167 9516 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:55:15.061707 10009 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:15.062141 10009 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:15.063529 10009 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:15.067310 8937 ts_manager.cc:194] Re-registered known tserver with Master: e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389)
I20250113 08:55:15.069253 8937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.1:36301
I20250113 08:55:15.082015 9902 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Log is configured to *not* fsync() on all Append() calls
W20250113 08:55:15.108376 9785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:55:15.130586 9881 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:55:15.131021 9881 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:15.133548 9881 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:15.169416 9344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 151 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
I20250113 08:55:15.169775 9344 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8ea89390544489a231c733997d2d9a in term 2.
I20250113 08:55:15.171386 9760 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab; no voters:
I20250113 08:55:15.171865 9881 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250113 08:55:15.172042 9881 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:55:15.172142 9881 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:55:15.172902 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 9393
I20250113 08:55:15.170503 9964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 151 } } ignore_live_leader: false dest_uuid: "e3f389d6e3484a13af0780989a117f02" is_pre_election: true
W20250113 08:55:15.177327 9760 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:55:15.179864 9881 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:15.180864 9881 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 3 election: Requested vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:15.181587 9344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 151 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab"
I20250113 08:55:15.181594 9964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 151 } } ignore_live_leader: false dest_uuid: "e3f389d6e3484a13af0780989a117f02"
I20250113 08:55:15.181872 9344 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:55:15.182277 9760 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:55:15.184298 9344 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8ea89390544489a231c733997d2d9a in term 3.
I20250113 08:55:15.185022 9760 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab; no voters: e3f389d6e3484a13af0780989a117f02
I20250113 08:55:15.185439 9881 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:55:15.186053 9881 raft_consensus.cc:695] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 3 LEADER]: Becoming Leader. State: Replica: 5f8ea89390544489a231c733997d2d9a, State: Running, Role: LEADER
I20250113 08:55:15.186852 9881 consensus_queue.cc:237] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 1.151, Last appended by leader: 151, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
W20250113 08:55:15.193173 9303 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56864: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:15.193152 8937 catalog_manager.cc:5526] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a reported cstate change: term changed from 1 to 3, leader changed from e3f389d6e3484a13af0780989a117f02 (127.7.123.1) to 5f8ea89390544489a231c733997d2d9a (127.7.123.2). New cstate: current_term: 3 leader_uuid: "5f8ea89390544489a231c733997d2d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } health_report { overall_health: UNKNOWN } } }
I20250113 08:55:15.201328 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.4:34591
--local_ip_for_outbound_sockets=127.7.123.4
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=45007
--webserver_interface=127.7.123.4
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250113 08:55:15.285596 9344 raft_consensus.cc:1270] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 3 FOLLOWER]: Refusing update from remote peer 5f8ea89390544489a231c733997d2d9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 3 index: 153. (index mismatch)
I20250113 08:55:15.286830 10021 consensus_queue.cc:1035] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 152, Last known committed idx: 146, Time since last communication: 0.000s
W20250113 08:55:15.287454 9760 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389): Couldn't send request to peer e3f389d6e3484a13af0780989a117f02. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:55:15.354219 10028 mvcc.cc:204] Tried to move back new op lower bound from 7113762878597615616 to 7113762878211276800. Current Snapshot: MvccSnapshot[applied={T|T < 7113762878597615616}]
W20250113 08:55:15.413126 9760 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.123.3:43245, user_credentials={real_user=slave}} blocked reactor thread for 48725.9us
W20250113 08:55:15.660082 9562 scanner-internal.cc:458] Time spent opening tablet: real 4.058s user 0.011s sys 0.000s
W20250113 08:55:15.698691 9564 scanner-internal.cc:458] Time spent opening tablet: real 4.056s user 0.010s sys 0.000s
I20250113 08:55:15.763464 9902 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=151 overwritten=0 applied=149 ignored=0} inserts{seen=7400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 08:55:15.764684 9902 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap complete.
I20250113 08:55:15.766914 9902 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Time spent bootstrapping tablet: real 0.791s user 0.699s sys 0.071s
I20250113 08:55:15.771834 9902 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 2 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
W20250113 08:55:15.817814 9760 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389): Couldn't send request to peer e3f389d6e3484a13af0780989a117f02. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:15.888382 9563 scanner-internal.cc:458] Time spent opening tablet: real 4.137s user 0.008s sys 0.004s
I20250113 08:55:15.896386 9902 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f389d6e3484a13af0780989a117f02, State: Initialized, Role: FOLLOWER
W20250113 08:55:15.897083 10027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:15.897651 10027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20250113 08:55:15.897639 9902 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 149, Last appended: 1.151, Last appended by leader: 151, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
W20250113 08:55:15.897895 10027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250113 08:55:15.906597 9902 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Time spent starting tablet: real 0.139s user 0.061s sys 0.071s
I20250113 08:55:15.907172 10009 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:55:15.908329 9964 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 2 FOLLOWER]: Advancing to term 3
W20250113 08:55:15.911938 10027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:15.912431 10027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.4
I20250113 08:55:15.931180 10027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.4:34591
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.7.123.4
--webserver_port=45007
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.4
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:15.934269 10027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:15.937978 10027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:16.147837 10051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:16.152549 10053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:16.168407 10027 server_base.cc:1034] running on GCE node
W20250113 08:55:16.187417 10050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:16.188719 10027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:16.190616 10027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:16.194120 10027 hybrid_clock.cc:648] HybridClock initialized: now 1736758516193959 us; error 107 us; skew 500 ppm
I20250113 08:55:16.194725 10027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:16.240419 10027 webserver.cc:458] Webserver started at http://127.7.123.4:45007/ using document root <none> and password file <none>
I20250113 08:55:16.241976 10027 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:16.242213 10027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:16.271034 10027 fs_manager.cc:714] Time spent opening directory manager: real 0.021s user 0.010s sys 0.000s
I20250113 08:55:16.277172 10046 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5f8ea89390544489a231c733997d2d9a)
I20250113 08:55:16.277556 10046 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:16.290645 10046 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:16.406045 10063 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:16.412626 10027 fs_manager.cc:730] Time spent opening block manager: real 0.139s user 0.013s sys 0.047s
I20250113 08:55:16.413020 10027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
uuid: "00524bc1b1a34b61af38c9f2bded4031"
format_stamp: "Formatted at 2025-01-13 08:55:06 on dist-test-slave-vxbw"
I20250113 08:55:16.414793 10027 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:16.442306 9825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 4 candidate_status { last_received { term: 3 index: 188 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a" is_pre_election: true
I20250113 08:55:16.455015 9344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 4 candidate_status { last_received { term: 3 index: 188 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
I20250113 08:55:16.458093 9899 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3f389d6e3484a13af0780989a117f02; no voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab
I20250113 08:55:16.459667 10046 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:55:16.509153 10027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:16.510622 10027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:16.511059 10027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:16.519362 10027 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:16.538849 10042 mvcc.cc:204] Tried to move back new op lower bound from 7113762880877711360 to 7113762878211276800. Current Snapshot: MvccSnapshot[applied={T|T < 7113762880877711360}]
I20250113 08:55:16.630497 10027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:55:16.631017 10027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20250113 08:55:16.631253 10027 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:55:16.631408 10027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:18.193009 10027 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.4:34591
I20250113 08:55:18.195622 10027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
I20250113 08:55:18.196660 10182 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.4:34591 every 8 connection(s)
I20250113 08:55:18.197050 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 10027
I20250113 08:55:18.197387 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 9261
I20250113 08:55:18.217157 9384 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:55:18.259660 10183 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:18.260125 10183 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:18.261602 10183 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:18.269119 8937 ts_manager.cc:194] Re-registered known tserver with Master: 00524bc1b1a34b61af38c9f2bded4031 (127.7.123.4:34591)
I20250113 08:55:18.270543 8937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.4:46423
I20250113 08:55:18.554405 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 9261
W20250113 08:55:18.621702 9534 connection.cc:537] client connection to 127.7.123.3:43245 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:55:18.622071 9562 meta_cache.cc:1510] marking tablet server cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) as failed
W20250113 08:55:18.622223 9760 connection.cc:537] client connection to 127.7.123.3:43245 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:55:18.622222 9562 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) has failed: Network error: TS failed: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:55:18.622493 9760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:55:18.623570 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.3:43245
--local_ip_for_outbound_sockets=127.7.123.3
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=35523
--webserver_interface=127.7.123.3
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:55:18.643124 9760 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.3:43245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:19.023402 9760 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.3:43245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250113 08:55:19.276741 10183 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
W20250113 08:55:19.314975 10193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:19.315697 10193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:19.315930 10193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:19.329816 10193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:19.330322 10193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.3
I20250113 08:55:19.347769 10193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.3:43245
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.123.3
--webserver_port=35523
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.3
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:19.350703 10193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:19.353569 10193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:19.377195 10201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:19.389304 10202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:19.398901 10193 server_base.cc:1034] running on GCE node
W20250113 08:55:19.400534 10206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:19.409826 10193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:19.420038 10193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:19.425910 10193 hybrid_clock.cc:648] HybridClock initialized: now 1736758519425788 us; error 95 us; skew 500 ppm
I20250113 08:55:19.426532 10193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:19.460057 10193 webserver.cc:458] Webserver started at http://127.7.123.3:35523/ using document root <none> and password file <none>
I20250113 08:55:19.461542 10193 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:19.464533 10193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:19.494288 10193 fs_manager.cc:714] Time spent opening directory manager: real 0.027s user 0.003s sys 0.001s
I20250113 08:55:19.520684 10211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:19.525590 10193 fs_manager.cc:730] Time spent opening block manager: real 0.025s user 0.007s sys 0.008s
I20250113 08:55:19.525846 10193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
uuid: "cf369533d8ad453f9d6c47cad3706dab"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:55:19.526961 10193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:19.563112 10193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:19.564538 10193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:19.564996 10193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:19.566530 10193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20250113 08:55:19.572259 9760 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.3:43245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250113 08:55:19.582296 10218 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:55:19.591534 10193 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:55:19.591735 10193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.007s
I20250113 08:55:19.591895 10193 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:55:19.596204 10193 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:55:19.596427 10193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.001s
I20250113 08:55:19.598038 10218 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap starting.
I20250113 08:55:19.922202 10218 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Log is configured to *not* fsync() on all Append() calls
I20250113 08:55:20.063246 10325 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.3:43245 every 8 connection(s)
I20250113 08:55:20.065773 10193 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.3:43245
I20250113 08:55:20.068347 10193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
I20250113 08:55:20.071918 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 10193
W20250113 08:55:20.145972 9760 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
I20250113 08:55:20.192590 10326 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:20.193058 10326 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:20.194588 10326 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:20.198640 8937 ts_manager.cc:194] Re-registered known tserver with Master: cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:20.200476 8937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.3:39471
W20250113 08:55:20.559355 9760 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
W20250113 08:55:21.098639 9760 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250113 08:55:21.213250 10326 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
W20250113 08:55:21.612260 9760 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250113 08:55:21.815241 10118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:21.831848 9805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:55:21.837280 10260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:21.853075 9944 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
W20250113 08:55:22.120352 9760 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250113 08:55:22.457962 10218 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap replayed 1/1 log segments. Stats: ops{read=338 overwritten=0 applied=336 ignored=0} inserts{seen=16700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 08:55:22.458936 10218 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap complete.
I20250113 08:55:22.462590 10218 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Time spent bootstrapping tablet: real 2.865s user 1.778s sys 0.157s
I20250113 08:55:22.475044 10218 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 3 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:22.488431 10218 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf369533d8ad453f9d6c47cad3706dab, State: Initialized, Role: FOLLOWER
I20250113 08:55:22.491605 10218 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 336, Last appended: 3.338, Last appended by leader: 338, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:22.499575 10218 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Time spent starting tablet: real 0.036s user 0.017s sys 0.004s
I20250113 08:55:29.236726 10260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:29.240798 10118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:29.253768 9944 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:55:29.253904 9805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:55:30.936287 8937 ts_manager.cc:281] Unset tserver state for e3f389d6e3484a13af0780989a117f02 from MAINTENANCE_MODE
I20250113 08:55:30.941721 9870 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:31.126139 8931 ts_manager.cc:281] Unset tserver state for 5f8ea89390544489a231c733997d2d9a from MAINTENANCE_MODE
I20250113 08:55:31.319113 10183 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:31.368364 8931 ts_manager.cc:281] Unset tserver state for 00524bc1b1a34b61af38c9f2bded4031 from MAINTENANCE_MODE
I20250113 08:55:31.395824 10009 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:31.834941 10326 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:31.954891 9870 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:32.327739 10183 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:33.195328 8931 ts_manager.cc:281] Unset tserver state for cf369533d8ad453f9d6c47cad3706dab from MAINTENANCE_MODE
I20250113 08:55:33.334044 10183 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:33.409341 10009 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:33.846280 10326 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:33.978665 9870 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:36.686962 8931 ts_manager.cc:292] Set tserver state for 5f8ea89390544489a231c733997d2d9a to MAINTENANCE_MODE
I20250113 08:55:36.941771 8931 ts_manager.cc:292] Set tserver state for 00524bc1b1a34b61af38c9f2bded4031 to MAINTENANCE_MODE
I20250113 08:55:37.443840 8931 ts_manager.cc:292] Set tserver state for e3f389d6e3484a13af0780989a117f02 to MAINTENANCE_MODE
I20250113 08:55:37.679086 8931 ts_manager.cc:292] Set tserver state for cf369533d8ad453f9d6c47cad3706dab to MAINTENANCE_MODE
I20250113 08:55:40.098062 10260 tablet_service.cc:1422] Tablet server cf369533d8ad453f9d6c47cad3706dab set to quiescing
I20250113 08:55:40.098356 10260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:41.178791 9805 tablet_service.cc:1422] Tablet server 5f8ea89390544489a231c733997d2d9a set to quiescing
I20250113 08:55:41.179131 9805 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:55:41.290443 10581 raft_consensus.cc:988] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: : Instructing follower e3f389d6e3484a13af0780989a117f02 to start an election
I20250113 08:55:41.290782 10581 raft_consensus.cc:1076] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 3 LEADER]: Signalling peer e3f389d6e3484a13af0780989a117f02 to start an election
I20250113 08:55:41.293191 9964 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a"
dest_uuid: "e3f389d6e3484a13af0780989a117f02"
from {username='slave'} at 127.7.123.2:49537
I20250113 08:55:41.293560 9964 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:55:41.293699 9964 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:55:41.298372 9964 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:41.301803 9964 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 4 election: Requested vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:41.303300 9825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 4 candidate_status { last_received { term: 3 index: 1752 } } ignore_live_leader: true dest_uuid: "5f8ea89390544489a231c733997d2d9a"
I20250113 08:55:41.303646 9825 raft_consensus.cc:3049] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:55:41.303755 9825 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5f8ea89390544489a231c733997d2d9a, State: Running, Role: LEADER
I20250113 08:55:41.304095 9825 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1752, Committed index: 1752, Last appended: 3.1754, Last appended by leader: 1754, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:41.309209 9825 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:55:41.313534 9825 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e3f389d6e3484a13af0780989a117f02 for term 4 because replica has last-logged OpId of term: 3 index: 1754, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1752.
W20250113 08:55:41.317409 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:41.327186 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:41.339125 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:41.353078 9923 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:41.399788 10279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 4 candidate_status { last_received { term: 3 index: 1752 } } ignore_live_leader: true dest_uuid: "cf369533d8ad453f9d6c47cad3706dab"
I20250113 08:55:41.400306 10279 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:55:41.409981 10279 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e3f389d6e3484a13af0780989a117f02 for term 4 because replica has last-logged OpId of term: 3 index: 1754, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1752.
I20250113 08:55:41.414580 9899 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3f389d6e3484a13af0780989a117f02; no voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab
W20250113 08:55:41.415731 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:41.428280 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:55:41.430586 10626 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250113 08:55:41.442494 9919 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:41.453202 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:41.471205 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:55:41.472187 10118 tablet_service.cc:1422] Tablet server 00524bc1b1a34b61af38c9f2bded4031 set to quiescing
I20250113 08:55:41.472482 10118 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:41.488139 9919 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:41.504163 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:41.522594 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:41.538264 9919 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:41.559123 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:41.583760 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:41.609431 9924 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:41.615401 10628 raft_consensus.cc:668] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: failed to trigger leader election: Illegal state: leader elections are disabled
W20250113 08:55:41.645695 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:41.662338 10581 raft_consensus.cc:668] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: failed to trigger leader election: Illegal state: leader elections are disabled
W20250113 08:55:41.676558 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:41.705524 9924 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:41.731401 10626 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:55:41.731670 10626 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:41.732612 10626 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:41.733578 9825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 5 candidate_status { last_received { term: 3 index: 1752 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a" is_pre_election: true
I20250113 08:55:41.733680 10279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 5 candidate_status { last_received { term: 3 index: 1752 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
I20250113 08:55:41.733968 9825 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3f389d6e3484a13af0780989a117f02 for term 5 because replica has last-logged OpId of term: 3 index: 1754, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1752.
I20250113 08:55:41.734086 10279 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3f389d6e3484a13af0780989a117f02 for term 5 because replica has last-logged OpId of term: 3 index: 1754, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1752.
I20250113 08:55:41.734750 9898 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3f389d6e3484a13af0780989a117f02; no voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab
I20250113 08:55:41.735147 10626 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:55:41.738267 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:41.772307 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:41.802656 9923 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:41.830169 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:41.863782 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:41.894898 9924 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:41.934217 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:41.974624 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:42.016533 9919 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:42.053637 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:42.097193 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:55:42.127485 10626 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:55:42.127911 10626 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:42.129097 10626 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:42.130213 10279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 5 candidate_status { last_received { term: 3 index: 1752 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
I20250113 08:55:42.131110 10279 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3f389d6e3484a13af0780989a117f02 for term 5 because replica has last-logged OpId of term: 3 index: 1754, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1752.
I20250113 08:55:42.133505 9825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 5 candidate_status { last_received { term: 3 index: 1752 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a" is_pre_election: true
I20250113 08:55:42.134073 9825 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3f389d6e3484a13af0780989a117f02 for term 5 because replica has last-logged OpId of term: 3 index: 1754, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1752.
I20250113 08:55:42.136797 9898 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3f389d6e3484a13af0780989a117f02; no voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab
I20250113 08:55:42.138216 10626 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:55:42.142330 9924 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:42.179360 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:42.223222 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:42.263072 9920 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:42.302271 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:42.346757 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:42.395249 9919 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:42.440688 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:42.499715 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:55:42.526384 10626 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:55:42.526759 10626 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:42.527971 10626 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:42.529328 10279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 5 candidate_status { last_received { term: 3 index: 1752 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
I20250113 08:55:42.529927 10279 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3f389d6e3484a13af0780989a117f02 for term 5 because replica has last-logged OpId of term: 3 index: 1754, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1752.
I20250113 08:55:42.532407 9825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 5 candidate_status { last_received { term: 3 index: 1752 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a" is_pre_election: true
I20250113 08:55:42.532980 9825 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3f389d6e3484a13af0780989a117f02 for term 5 because replica has last-logged OpId of term: 3 index: 1754, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1752.
I20250113 08:55:42.533939 9898 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3f389d6e3484a13af0780989a117f02; no voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab
I20250113 08:55:42.534575 10626 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:55:42.550642 9919 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:42.600741 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:42.656560 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:42.705996 9919 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:42.711524 9944 tablet_service.cc:1422] Tablet server e3f389d6e3484a13af0780989a117f02 set to quiescing
I20250113 08:55:42.711856 9944 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
W20250113 08:55:42.756832 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:42.811015 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:42.864660 9919 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:42.916903 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:42.974951 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:43.027844 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:43.096788 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:43.156581 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:43.214006 9920 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:43.271821 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:43.338501 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:43.350258 10643 raft_consensus.cc:668] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: failed to trigger leader election: Illegal state: leader elections are disabled
W20250113 08:55:43.398341 9920 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:43.464830 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:43.533070 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:43.595327 9919 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:43.658401 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:43.713812 9805 tablet_service.cc:1422] Tablet server 5f8ea89390544489a231c733997d2d9a set to quiescing
I20250113 08:55:43.714305 9805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:43.723557 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:43.790544 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:43.854907 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:43.926764 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:43.995446 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:44.065883 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:44.139101 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:44.210650 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:44.282768 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:44.358389 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:44.433215 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:44.509361 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:44.589119 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:44.667492 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:44.748312 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:44.828533 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:44.910001 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:44.993489 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:45.075878 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:45.160612 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:45.245314 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:45.338263 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:45.427641 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:45.513142 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:45.601981 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:55:45.690392 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:45.781519 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:45.852385 9944 tablet_service.cc:1422] Tablet server e3f389d6e3484a13af0780989a117f02 set to quiescing
I20250113 08:55:45.852620 9944 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:45.880414 9781 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:55:45.951092 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 9741
I20250113 08:55:45.961105 9865 generic_service.cc:196] Checking for leaks (request via RPC)
W20250113 08:55:45.975373 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:46.071236 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:46.262966 9784 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41224: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:55:46.263700 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 9741
W20250113 08:55:46.325213 9533 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333) has failed: Network error: recv got EOF from 127.7.123.2:43333 (error 108)
W20250113 08:55:46.325235 8923 connection.cc:537] server connection from 127.7.123.2:34725 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:55:46.325892 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.2:43333
--local_ip_for_outbound_sockets=127.7.123.2
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=39675
--webserver_interface=127.7.123.2
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:55:46.342629 9919 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:46.342629 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:46.422155 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:46.520246 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:46.716785 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:46.815519 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:46.825325 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:46.825795 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:46.825922 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:46.835479 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:46.835988 10670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.2
I20250113 08:55:46.847770 10670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.2:43333
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.123.2
--webserver_port=39675
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.2
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:46.849792 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:46.852421 10670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:46.871377 10676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:46.872898 10677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:46.875555 10679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:46.877518 10670 server_base.cc:1034] running on GCE node
I20250113 08:55:46.878473 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:46.880126 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:46.881646 10670 hybrid_clock.cc:648] HybridClock initialized: now 1736758546881554 us; error 61 us; skew 500 ppm
I20250113 08:55:46.882093 10670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:46.885725 10670 webserver.cc:458] Webserver started at http://127.7.123.2:39675/ using document root <none> and password file <none>
I20250113 08:55:46.886778 10670 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:46.887004 10670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:46.893982 10670 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20250113 08:55:46.898686 10685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:46.901279 10670 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20250113 08:55:46.901564 10670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
uuid: "5f8ea89390544489a231c733997d2d9a"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:55:46.902727 10670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:46.940048 10670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:46.941499 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:46.942025 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:46.943653 10670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:46.946460 10692 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:55:46.954327 10670 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:55:46.954521 10670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20250113 08:55:46.955109 10670 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:55:46.959192 10670 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:55:46.959322 10670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20250113 08:55:46.959920 10692 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap starting.
I20250113 08:55:47.017299 10670 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.2:43333
I20250113 08:55:47.017308 10798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.2:43333 every 8 connection(s)
I20250113 08:55:47.020117 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
W20250113 08:55:47.020083 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:47.030499 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 10670
I20250113 08:55:47.030789 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 9878
I20250113 08:55:47.077814 10799 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:47.078233 10799 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:47.080221 10799 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:47.089411 10003 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:55:47.089577 8931 ts_manager.cc:194] Re-registered known tserver with Master: 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333)
I20250113 08:55:47.091524 8931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.2:33397
W20250113 08:55:47.122462 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:47.362155 9921 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53186: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:55:47.362339 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 9878
W20250113 08:55:47.424641 9534 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389) has failed: Network error: recv got EOF from 127.7.123.1:38389 (error 108)
I20250113 08:55:47.425485 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.1:38389
--local_ip_for_outbound_sockets=127.7.123.1
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=33665
--webserver_interface=127.7.123.1
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:55:47.530388 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:47.660673 10692 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Log is configured to *not* fsync() on all Append() calls
W20250113 08:55:47.854979 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:47.992964 10807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:47.993495 10807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:47.993954 10807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:48.006538 10807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:48.006862 10807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.1
I20250113 08:55:48.024019 10807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.1:38389
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.123.1
--webserver_port=33665
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.1
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:48.026427 10807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:48.029296 10807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:48.064105 10815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:48.064851 10817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:48.065250 10814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:48.067076 10807 server_base.cc:1034] running on GCE node
I20250113 08:55:48.067924 10807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:48.069542 10807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:48.071237 10807 hybrid_clock.cc:648] HybridClock initialized: now 1736758548071119 us; error 101 us; skew 500 ppm
I20250113 08:55:48.071696 10807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:48.081040 10807 webserver.cc:458] Webserver started at http://127.7.123.1:33665/ using document root <none> and password file <none>
I20250113 08:55:48.083065 10807 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:48.083256 10807 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:48.094101 10807 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.009s sys 0.000s
I20250113 08:55:48.101742 10823 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:48.103152 10799 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:55:48.103863 10807 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
I20250113 08:55:48.104082 10807 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
uuid: "e3f389d6e3484a13af0780989a117f02"
format_stamp: "Formatted at 2025-01-13 08:55:04 on dist-test-slave-vxbw"
I20250113 08:55:48.105612 10807 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:48.129289 10807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:48.130563 10807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:48.131007 10807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:48.132391 10807 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:48.137907 10830 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:55:48.147505 10807 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:55:48.147763 10807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250113 08:55:48.148288 10807 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:55:48.155098 10807 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:55:48.155262 10807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.008s sys 0.000s
I20250113 08:55:48.155978 10830 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap starting.
W20250113 08:55:48.186295 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:48.227612 10807 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.1:38389
I20250113 08:55:48.227664 10936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.1:38389 every 8 connection(s)
I20250113 08:55:48.230227 10807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
I20250113 08:55:48.232998 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 10807
I20250113 08:55:48.233304 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 10027
I20250113 08:55:48.271476 10178 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:55:48.279249 10937 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:48.279896 10937 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:48.281384 10937 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:48.285408 8931 ts_manager.cc:194] Re-registered known tserver with Master: e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389)
I20250113 08:55:48.287218 8931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.1:41281
I20250113 08:55:48.444232 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 10027
I20250113 08:55:48.468417 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.4:34591
--local_ip_for_outbound_sockets=127.7.123.4
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=45007
--webserver_interface=127.7.123.4
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:55:48.557078 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:55:48.911011 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:48.968237 10830 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Log is configured to *not* fsync() on all Append() calls
W20250113 08:55:49.001360 10945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:49.001958 10945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:49.002218 10945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:49.017568 10945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:49.018003 10945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.4
I20250113 08:55:49.035651 10945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.4:34591
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.7.123.4
--webserver_port=45007
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.4
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:49.038312 10945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:49.041697 10945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:49.070149 10951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:49.071046 10952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:49.081458 10945 server_base.cc:1034] running on GCE node
W20250113 08:55:49.082139 10954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:49.083257 10945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:49.085019 10945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:49.086604 10945 hybrid_clock.cc:648] HybridClock initialized: now 1736758549086459 us; error 71 us; skew 500 ppm
I20250113 08:55:49.087195 10945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:49.097622 10945 webserver.cc:458] Webserver started at http://127.7.123.4:45007/ using document root <none> and password file <none>
I20250113 08:55:49.099143 10945 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:49.099434 10945 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:49.111101 10945 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.009s sys 0.000s
I20250113 08:55:49.116398 10960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:49.119153 10945 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20250113 08:55:49.119479 10945 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
uuid: "00524bc1b1a34b61af38c9f2bded4031"
format_stamp: "Formatted at 2025-01-13 08:55:06 on dist-test-slave-vxbw"
I20250113 08:55:49.120992 10945 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:49.150909 10945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:49.152643 10945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:49.153286 10945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:49.161970 10945 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:49.164860 10945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:55:49.165060 10945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:49.165212 10945 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:55:49.165298 10945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:49.220536 10945 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.4:34591
I20250113 08:55:49.220593 11072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.4:34591 every 8 connection(s)
I20250113 08:55:49.223315 10945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
I20250113 08:55:49.230707 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 10945
I20250113 08:55:49.231508 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 10193
W20250113 08:55:49.274319 10240 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42690: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:55:49.281791 10319 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:55:49.284957 11073 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:49.285424 11073 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:49.287148 11073 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:49.290163 8931 ts_manager.cc:194] Re-registered known tserver with Master: 00524bc1b1a34b61af38c9f2bded4031 (127.7.123.4:34591)
I20250113 08:55:49.291663 8931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.4:57787
I20250113 08:55:49.292618 10937 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:55:49.554932 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 10193
W20250113 08:55:49.614118 9534 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) has failed: Network error: recv got EOF from 127.7.123.3:43245 (error 108)
I20250113 08:55:49.615945 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.3:43245
--local_ip_for_outbound_sockets=127.7.123.3
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=35523
--webserver_interface=127.7.123.3
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250113 08:55:49.749960 9564 meta_cache.cc:1510] marking tablet server cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) as failed
I20250113 08:55:49.772773 9562 meta_cache.cc:1510] marking tablet server cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) as failed
W20250113 08:55:50.294603 11080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:50.295179 11080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:50.295372 11080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250113 08:55:50.299695 11073 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
W20250113 08:55:50.312028 11080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:50.312402 11080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.3
I20250113 08:55:50.330224 11080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.3:43245
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.123.3
--webserver_port=35523
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.3
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:50.333096 11080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:50.336086 11080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:50.356379 11086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:50.358379 11087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:50.360725 11089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:50.362301 11080 server_base.cc:1034] running on GCE node
I20250113 08:55:50.364094 11080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:50.365664 11080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:50.367274 11080 hybrid_clock.cc:648] HybridClock initialized: now 1736758550367151 us; error 80 us; skew 500 ppm
I20250113 08:55:50.367782 11080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:50.376848 11080 webserver.cc:458] Webserver started at http://127.7.123.3:35523/ using document root <none> and password file <none>
I20250113 08:55:50.378342 11080 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:50.378551 11080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:50.388325 11080 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20250113 08:55:50.394963 11095 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:50.398265 11080 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20250113 08:55:50.398584 11080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
uuid: "cf369533d8ad453f9d6c47cad3706dab"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:55:50.400211 11080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:50.425993 11080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:50.427711 11080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:50.428225 11080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:50.431399 11080 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:50.439332 11102 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:55:50.448590 11080 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:55:50.448920 11080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.002s
I20250113 08:55:50.449144 11080 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:55:50.454552 11080 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:55:50.454787 11080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.004s sys 0.002s
I20250113 08:55:50.455355 11102 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap starting.
W20250113 08:55:50.649566 9534 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) has failed: Network error: Client connection negotiation failed: client connection to 127.7.123.3:43245: connect: Connection refused (error 111)
I20250113 08:55:51.368577 11208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.3:43245 every 8 connection(s)
I20250113 08:55:51.369077 11080 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.3:43245
I20250113 08:55:51.371882 11080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
I20250113 08:55:51.379962 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 11080
I20250113 08:55:51.453846 11209 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:55:51.454317 11209 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:51.455960 11209 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:55:51.459904 8937 ts_manager.cc:194] Re-registered known tserver with Master: cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:51.461726 8937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.3:49111
I20250113 08:55:51.546665 11102 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Log is configured to *not* fsync() on all Append() calls
I20250113 08:55:52.466365 11209 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:55:52.666311 10692 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap replaying log segment 1/1 (1.76M/3.03M this segment, stats: ops{read=1020 overwritten=0 applied=1019 ignored=0} inserts{seen=50850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250113 08:55:52.855687 11008 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:52.881644 10872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:52.885170 10734 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:52.892432 11144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:53.971694 10830 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap replaying log segment 1/1 (1.73M/2.89M this segment, stats: ops{read=1050 overwritten=0 applied=1050 ignored=0} inserts{seen=52400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250113 08:55:54.564159 9564 scanner-internal.cc:172] Scan of tablet b13658abfdf04a8c9fcc1f08c15ff92a at cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) deadline exceeded after 10.000s
W20250113 08:55:54.564611 9564 scanner-internal.cc:458] Time spent opening tablet: real 10.555s user 0.017s sys 0.001s
I20250113 08:55:54.778141 9563 scanner-internal.cc:172] Scan of tablet b13658abfdf04a8c9fcc1f08c15ff92a at cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) deadline exceeded after 10.000s
W20250113 08:55:54.778371 9563 scanner-internal.cc:458] Time spent opening tablet: real 11.583s user 0.014s sys 0.006s
I20250113 08:55:54.792634 9562 scanner-internal.cc:172] Scan of tablet b13658abfdf04a8c9fcc1f08c15ff92a at cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) deadline exceeded after 10.000s
W20250113 08:55:54.792862 9562 scanner-internal.cc:458] Time spent opening tablet: real 11.139s user 0.018s sys 0.001s
I20250113 08:55:55.854584 10692 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1754 overwritten=0 applied=1752 ignored=0} inserts{seen=87500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 08:55:55.855571 10692 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap complete.
I20250113 08:55:55.857836 10692 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Time spent bootstrapping tablet: real 8.898s user 8.007s sys 0.632s
I20250113 08:55:55.866477 10692 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:55.881922 10692 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8ea89390544489a231c733997d2d9a, State: Initialized, Role: FOLLOWER
I20250113 08:55:55.883374 10692 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1752, Last appended: 3.1754, Last appended by leader: 1754, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:55.888420 10692 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Time spent starting tablet: real 0.030s user 0.025s sys 0.007s
I20250113 08:55:56.262377 11251 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:55:56.262758 11251 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:56.265286 11251 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:56.294797 10892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 1754 } } ignore_live_leader: false dest_uuid: "e3f389d6e3484a13af0780989a117f02" is_pre_election: true
W20250113 08:55:56.300868 10689 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:55:56.306612 10713 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:55:56.304744 11164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 1754 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
W20250113 08:55:56.310130 10689 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:55:56.310433 10689 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a; no voters: cf369533d8ad453f9d6c47cad3706dab, e3f389d6e3484a13af0780989a117f02
I20250113 08:55:56.310994 11251 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250113 08:55:56.558980 11102 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap replaying log segment 1/1 (1.70M/2.89M this segment, stats: ops{read=1051 overwritten=0 applied=1050 ignored=0} inserts{seen=52400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250113 08:55:56.757428 11251 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:55:56.757673 11251 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:56.758435 11251 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:56.759727 10892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 1754 } } ignore_live_leader: false dest_uuid: "e3f389d6e3484a13af0780989a117f02" is_pre_election: true
W20250113 08:55:56.762167 10689 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:55:56.762758 11164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 1754 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
W20250113 08:55:56.763715 10689 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:55:56.763911 10689 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a; no voters: cf369533d8ad453f9d6c47cad3706dab, e3f389d6e3484a13af0780989a117f02
I20250113 08:55:56.764282 11251 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250113 08:55:56.824401 10713 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:55:57.028543 10830 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=1752 overwritten=0 applied=1752 ignored=0} inserts{seen=87500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:55:57.029244 10830 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap complete.
I20250113 08:55:57.031016 10830 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Time spent bootstrapping tablet: real 8.876s user 8.176s sys 0.575s
I20250113 08:55:57.036800 10830 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:57.037853 10830 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f389d6e3484a13af0780989a117f02, State: Initialized, Role: FOLLOWER
I20250113 08:55:57.039827 10830 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1752, Last appended: 3.1752, Last appended by leader: 1752, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:57.042418 10830 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20250113 08:55:57.085923 11251 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:55:57.086196 11251 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:57.087366 11251 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:57.087858 10892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 1754 } } ignore_live_leader: false dest_uuid: "e3f389d6e3484a13af0780989a117f02" is_pre_election: true
I20250113 08:55:57.087967 11164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 1754 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
I20250113 08:55:57.088395 10892 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f8ea89390544489a231c733997d2d9a in term 4.
W20250113 08:55:57.088719 10689 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:55:57.088994 10689 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a, e3f389d6e3484a13af0780989a117f02; no voters: cf369533d8ad453f9d6c47cad3706dab
I20250113 08:55:57.089288 11251 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250113 08:55:57.089416 11251 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:55:57.089494 11251 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:55:57.093804 11251 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:57.094759 11251 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 election: Requested vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:55:57.095645 10892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 1754 } } ignore_live_leader: false dest_uuid: "e3f389d6e3484a13af0780989a117f02"
I20250113 08:55:57.095901 10892 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:55:57.100792 11164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 5 candidate_status { last_received { term: 3 index: 1754 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab"
I20250113 08:55:57.101034 10892 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f8ea89390544489a231c733997d2d9a in term 5.
W20250113 08:55:57.101744 10689 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 election: Tablet error from VoteRequest() call to peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:55:57.102196 10689 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a, e3f389d6e3484a13af0780989a117f02; no voters: cf369533d8ad453f9d6c47cad3706dab
I20250113 08:55:57.102665 11251 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 5 FOLLOWER]: Leader election won for term 5
I20250113 08:55:57.103380 11251 raft_consensus.cc:695] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 5 LEADER]: Becoming Leader. State: Replica: 5f8ea89390544489a231c733997d2d9a, State: Running, Role: LEADER
I20250113 08:55:57.104382 11251 consensus_queue.cc:237] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1752, Committed index: 1752, Last appended: 3.1754, Last appended by leader: 1754, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:55:57.111141 8931 catalog_manager.cc:5526] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "5f8ea89390544489a231c733997d2d9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } health_report { overall_health: UNKNOWN } } }
I20250113 08:55:57.197796 10892 raft_consensus.cc:1270] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 5 FOLLOWER]: Refusing update from remote peer 5f8ea89390544489a231c733997d2d9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 1752. Preceding OpId from leader: term: 5 index: 1755. (index mismatch)
I20250113 08:55:57.199407 11251 consensus_queue.cc:1035] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1755, Last known committed idx: 1752, Time since last communication: 0.000s
I20250113 08:55:57.216789 11265 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52884 (ReqId={client: 174b6e3e9f79481394e37ad662f9ad1a, seq_no=1751, attempt_no=107}) took 1112 ms. Trace:
I20250113 08:55:57.217182 11265 rpcz_store.cc:276] 0113 08:55:56.104534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:55:56.104618 (+ 84us) service_pool.cc:225] Handling call
0113 08:55:57.216688 (+1112070us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:55:57.224550 11265 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52884 (ReqId={client: 174b6e3e9f79481394e37ad662f9ad1a, seq_no=1750, attempt_no=107}) took 1119 ms. Trace:
W20250113 08:55:57.224984 10689 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:55:57.224903 11265 rpcz_store.cc:276] 0113 08:55:56.105233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:55:56.105364 (+ 131us) service_pool.cc:225] Handling call
0113 08:55:57.224459 (+1119095us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250113 08:55:57.237810 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:55:57.764752 10689 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:58.287451 10689 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250113 08:55:58.831912 10689 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
W20250113 08:55:59.346748 10689 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
I20250113 08:55:59.583529 11008 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:59.619681 10734 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:55:59.670327 11144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:59.673275 10872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:59.887459 10689 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250113 08:56:00.401355 11102 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap replayed 1/1 log segments. Stats: ops{read=1754 overwritten=0 applied=1752 ignored=0} inserts{seen=87500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 08:56:00.402451 11102 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap complete.
I20250113 08:56:00.408049 11102 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Time spent bootstrapping tablet: real 9.953s user 8.877s sys 0.570s
W20250113 08:56:00.411870 10689 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250113 08:56:00.419265 11102 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 4 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:00.427125 11102 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf369533d8ad453f9d6c47cad3706dab, State: Initialized, Role: FOLLOWER
I20250113 08:56:00.428745 11102 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1752, Last appended: 3.1754, Last appended by leader: 1754, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:00.432940 11102 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Time spent starting tablet: real 0.025s user 0.019s sys 0.000s
I20250113 08:56:00.521138 11164 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:56:01.452586 11314 mvcc.cc:204] Tried to move back new op lower bound from 7113763063726878720 to 7113763049905414144. Current Snapshot: MvccSnapshot[applied={T|T < 7113763063694950400}]
I20250113 08:56:06.416115 11008 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:06.460183 10734 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:56:06.482072 11144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:06.524434 10872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:56:06.985814 9533 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.7.123.2:43333, user_credentials={real_user=slave}} blocked reactor thread for 43572.9us
I20250113 08:56:08.210330 8930 ts_manager.cc:281] Unset tserver state for e3f389d6e3484a13af0780989a117f02 from MAINTENANCE_MODE
I20250113 08:56:08.281915 8930 ts_manager.cc:281] Unset tserver state for 5f8ea89390544489a231c733997d2d9a from MAINTENANCE_MODE
I20250113 08:56:08.409461 11073 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:08.410967 10937 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:08.564054 8930 ts_manager.cc:281] Unset tserver state for 00524bc1b1a34b61af38c9f2bded4031 from MAINTENANCE_MODE
I20250113 08:56:08.641207 8930 ts_manager.cc:281] Unset tserver state for cf369533d8ad453f9d6c47cad3706dab from MAINTENANCE_MODE
I20250113 08:56:08.743191 11209 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:08.793138 10799 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:09.421062 11073 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:09.458930 10937 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:12.758723 8930 ts_manager.cc:292] Set tserver state for 00524bc1b1a34b61af38c9f2bded4031 to MAINTENANCE_MODE
I20250113 08:56:12.831019 8930 ts_manager.cc:292] Set tserver state for e3f389d6e3484a13af0780989a117f02 to MAINTENANCE_MODE
I20250113 08:56:13.696346 8931 ts_manager.cc:292] Set tserver state for cf369533d8ad453f9d6c47cad3706dab to MAINTENANCE_MODE
I20250113 08:56:13.792673 8931 ts_manager.cc:292] Set tserver state for 5f8ea89390544489a231c733997d2d9a to MAINTENANCE_MODE
I20250113 08:56:16.046974 10734 tablet_service.cc:1422] Tablet server 5f8ea89390544489a231c733997d2d9a set to quiescing
I20250113 08:56:16.047389 10734 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:56:16.056082 11382 raft_consensus.cc:988] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: : Instructing follower e3f389d6e3484a13af0780989a117f02 to start an election
I20250113 08:56:16.056272 11382 raft_consensus.cc:1076] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 5 LEADER]: Signalling peer e3f389d6e3484a13af0780989a117f02 to start an election
I20250113 08:56:16.058005 10891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a"
dest_uuid: "e3f389d6e3484a13af0780989a117f02"
from {username='slave'} at 127.7.123.2:41887
I20250113 08:56:16.058373 10891 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:16.058511 10891 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:56:16.061543 10891 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:16.077095 10891 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 6 election: Requested vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:56:16.099558 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3052 } } ignore_live_leader: true dest_uuid: "cf369533d8ad453f9d6c47cad3706dab"
I20250113 08:56:16.100006 11162 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:56:16.103857 11162 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f389d6e3484a13af0780989a117f02 in term 6.
I20250113 08:56:16.105911 11162 raft_consensus.cc:1235] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 6 FOLLOWER]: Rejecting Update request from peer 5f8ea89390544489a231c733997d2d9a for earlier term 5. Current term is 6. Ops: [5.3053-5.3053]
I20250113 08:56:16.107187 11439 consensus_queue.cc:1046] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 }, Status: INVALID_TERM, Last received: 5.3052, Next index: 3053, Last known committed idx: 3052, Time since last communication: 0.000s
I20250113 08:56:16.107571 11439 raft_consensus.cc:3049] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 5 LEADER]: Stepping down as leader of term 5
I20250113 08:56:16.107652 11439 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5f8ea89390544489a231c733997d2d9a, State: Running, Role: LEADER
I20250113 08:56:16.107955 11439 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3052, Committed index: 3052, Last appended: 5.3053, Last appended by leader: 3053, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:16.109460 11439 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:56:16.115250 10827 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf369533d8ad453f9d6c47cad3706dab, e3f389d6e3484a13af0780989a117f02; no voters:
W20250113 08:56:16.115974 11269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.037s ago.
I20250113 08:56:16.117692 10754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 6 candidate_status { last_received { term: 5 index: 3052 } } ignore_live_leader: true dest_uuid: "5f8ea89390544489a231c733997d2d9a"
I20250113 08:56:16.131459 10754 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e3f389d6e3484a13af0780989a117f02 for term 6 because replica has last-logged OpId of term: 5 index: 3053, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3052.
I20250113 08:56:16.139281 11532 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 6 FOLLOWER]: Leader election won for term 6
W20250113 08:56:16.139773 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Service unavailable: leader is not yet ready
I20250113 08:56:16.150660 11532 raft_consensus.cc:695] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 6 LEADER]: Becoming Leader. State: Replica: e3f389d6e3484a13af0780989a117f02, State: Running, Role: LEADER
W20250113 08:56:16.150756 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Service unavailable: leader is not yet ready
I20250113 08:56:16.152386 11532 consensus_queue.cc:237] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3052, Committed index: 3052, Last appended: 5.3052, Last appended by leader: 3052, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:16.158223 8931 catalog_manager.cc:5526] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 reported cstate change: term changed from 5 to 6, leader changed from 5f8ea89390544489a231c733997d2d9a (127.7.123.2) to e3f389d6e3484a13af0780989a117f02 (127.7.123.1). New cstate: current_term: 6 leader_uuid: "e3f389d6e3484a13af0780989a117f02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:16.163719 11162 raft_consensus.cc:1270] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 6 FOLLOWER]: Refusing update from remote peer e3f389d6e3484a13af0780989a117f02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3052. Preceding OpId from leader: term: 6 index: 3054. (index mismatch)
I20250113 08:56:16.165104 11533 consensus_queue.cc:1035] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3053, Last known committed idx: 3052, Time since last communication: 0.000s
I20250113 08:56:16.164155 10754 raft_consensus.cc:1270] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 6 FOLLOWER]: Refusing update from remote peer e3f389d6e3484a13af0780989a117f02: Log matching property violated. Preceding OpId in replica: term: 5 index: 3053. Preceding OpId from leader: term: 6 index: 3054. (index mismatch)
I20250113 08:56:16.167707 11533 consensus_queue.cc:1035] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3053, Last known committed idx: 3052, Time since last communication: 0.001s
I20250113 08:56:16.168466 11533 consensus_queue.cc:1225] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Peer 5f8ea89390544489a231c733997d2d9a log is divergent from this leader: its last log entry 5.3053 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3052
I20250113 08:56:16.170892 10754 pending_rounds.cc:77] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Aborting all ops after (but not including) 3052
I20250113 08:56:16.171418 10754 pending_rounds.cc:99] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Aborting uncommitted WRITE_OP operation due to leader change: 5.3053
W20250113 08:56:16.171876 10754 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Aborted: Op aborted by new leader
I20250113 08:56:16.179863 11263 mvcc.cc:204] Tried to move back new op lower bound from 7113763127956500480 to 7113763127927316480. Current Snapshot: MvccSnapshot[applied={T|T < 7113763127956500480}]
I20250113 08:56:16.184844 11269 mvcc.cc:204] Tried to move back new op lower bound from 7113763127956500480 to 7113763127927316480. Current Snapshot: MvccSnapshot[applied={T|T < 7113763127956500480 or (T in {7113763127956500480})}]
I20250113 08:56:16.514822 10872 tablet_service.cc:1422] Tablet server e3f389d6e3484a13af0780989a117f02 set to quiescing
I20250113 08:56:16.515162 10872 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:56:16.535887 11532 raft_consensus.cc:988] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: : Instructing follower cf369533d8ad453f9d6c47cad3706dab to start an election
I20250113 08:56:16.536139 11532 raft_consensus.cc:1076] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 6 LEADER]: Signalling peer cf369533d8ad453f9d6c47cad3706dab to start an election
I20250113 08:56:16.537861 11163 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a"
dest_uuid: "cf369533d8ad453f9d6c47cad3706dab"
from {username='slave'} at 127.7.123.1:44461
I20250113 08:56:16.538298 11163 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:56:16.538440 11163 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:56:16.542018 11163 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:16.558776 11162 raft_consensus.cc:1235] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 7 FOLLOWER]: Rejecting Update request from peer e3f389d6e3484a13af0780989a117f02 for earlier term 6. Current term is 7. Ops: []
I20250113 08:56:16.560225 11532 consensus_queue.cc:1046] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 }, Status: INVALID_TERM, Last received: 6.3068, Next index: 3069, Last known committed idx: 3067, Time since last communication: 0.000s
I20250113 08:56:16.560750 11532 raft_consensus.cc:3049] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 6 LEADER]: Stepping down as leader of term 6
I20250113 08:56:16.560900 11532 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e3f389d6e3484a13af0780989a117f02, State: Running, Role: LEADER
I20250113 08:56:16.561317 11532 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3068, Committed index: 3068, Last appended: 6.3070, Last appended by leader: 3070, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:16.561722 11532 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:56:16.568434 11163 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 7 election: Requested vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333)
W20250113 08:56:16.581835 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:16.587913 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:16.594228 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:16.604954 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:56:16.610873 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "cf369533d8ad453f9d6c47cad3706dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 3068 } } ignore_live_leader: true dest_uuid: "e3f389d6e3484a13af0780989a117f02"
I20250113 08:56:16.611578 10891 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cf369533d8ad453f9d6c47cad3706dab for term 7 because replica has last-logged OpId of term: 6 index: 3070, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3068.
I20250113 08:56:16.612241 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "cf369533d8ad453f9d6c47cad3706dab" candidate_term: 7 candidate_status { last_received { term: 6 index: 3068 } } ignore_live_leader: true dest_uuid: "5f8ea89390544489a231c733997d2d9a"
I20250113 08:56:16.612476 10753 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 6 FOLLOWER]: Advancing to term 7
I20250113 08:56:16.615983 10753 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cf369533d8ad453f9d6c47cad3706dab for term 7 because replica has last-logged OpId of term: 6 index: 3070, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3068.
I20250113 08:56:16.617480 11099 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf369533d8ad453f9d6c47cad3706dab; no voters: 5f8ea89390544489a231c733997d2d9a, e3f389d6e3484a13af0780989a117f02
I20250113 08:56:16.618402 11535 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250113 08:56:16.619400 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:16.627833 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:16.642527 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:16.656109 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:16.670132 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:16.686108 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:16.700362 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:16.714385 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:16.734777 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:16.752789 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:16.774188 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:16.798740 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:16.820622 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:16.843953 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:16.857151 11562 raft_consensus.cc:668] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: failed to trigger leader election: Illegal state: leader elections are disabled
W20250113 08:56:16.873030 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:16.900135 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:56:16.906265 11008 tablet_service.cc:1422] Tablet server 00524bc1b1a34b61af38c9f2bded4031 set to quiescing
I20250113 08:56:16.906780 11008 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:56:16.911947 11532 raft_consensus.cc:668] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: failed to trigger leader election: Illegal state: leader elections are disabled
W20250113 08:56:16.928407 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:16.957338 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:16.986371 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:17.012331 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:17.046078 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:56:17.055126 11535 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:17.055408 11535 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:17.056396 11535 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333)
I20250113 08:56:17.057530 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "cf369533d8ad453f9d6c47cad3706dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 3068 } } ignore_live_leader: false dest_uuid: "e3f389d6e3484a13af0780989a117f02" is_pre_election: true
I20250113 08:56:17.057896 10891 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf369533d8ad453f9d6c47cad3706dab for term 8 because replica has last-logged OpId of term: 6 index: 3070, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3068.
I20250113 08:56:17.058316 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "cf369533d8ad453f9d6c47cad3706dab" candidate_term: 8 candidate_status { last_received { term: 6 index: 3068 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a" is_pre_election: true
I20250113 08:56:17.058729 10753 raft_consensus.cc:2405] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf369533d8ad453f9d6c47cad3706dab for term 8 because replica has last-logged OpId of term: 6 index: 3070, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3068.
I20250113 08:56:17.061584 11099 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf369533d8ad453f9d6c47cad3706dab; no voters: 5f8ea89390544489a231c733997d2d9a, e3f389d6e3484a13af0780989a117f02
I20250113 08:56:17.062115 11535 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250113 08:56:17.075392 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:17.107013 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:17.142931 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:17.175165 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:17.208622 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:17.247650 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:17.282918 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:56:17.299672 11144 tablet_service.cc:1422] Tablet server cf369533d8ad453f9d6c47cad3706dab set to quiescing
I20250113 08:56:17.300048 11144 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:56:17.318910 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:17.362984 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:17.400450 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:17.441680 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:17.477581 11535 raft_consensus.cc:668] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: failed to trigger leader election: Illegal state: leader elections are disabled
W20250113 08:56:17.486851 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:17.531494 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:17.580763 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:17.630980 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:17.677165 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:17.722576 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:17.774631 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:17.823594 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:17.872696 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:17.928324 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:17.980326 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:18.030174 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:18.086156 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:18.142496 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:56:18.174582 10734 tablet_service.cc:1422] Tablet server 5f8ea89390544489a231c733997d2d9a set to quiescing
I20250113 08:56:18.174942 10734 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:56:18.202373 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:18.259965 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:18.316987 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:18.376080 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:18.437878 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:18.496848 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:18.559393 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:18.626288 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:18.689674 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:18.753057 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:18.821621 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:18.887073 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:18.955837 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:19.028131 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:19.095374 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:19.164553 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:19.239111 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:19.314169 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:19.386463 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:19.465184 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:56:19.510895 10872 tablet_service.cc:1422] Tablet server e3f389d6e3484a13af0780989a117f02 set to quiescing
I20250113 08:56:19.511205 10872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:56:19.539904 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:56:19.609392 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 10670
W20250113 08:56:19.617367 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:19.620375 10793 generic_service.cc:196] Checking for leaks (request via RPC)
W20250113 08:56:19.696062 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:20.013782 10712 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52884: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:56:20.015102 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 10670
I20250113 08:56:20.081740 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.2:43333
--local_ip_for_outbound_sockets=127.7.123.2
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=39675
--webserver_interface=127.7.123.2
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:56:20.098737 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:20.181285 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:20.260377 9533 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333) has failed: Network error: Client connection negotiation failed: client connection to 127.7.123.2:43333: connect: Connection refused (error 111)
W20250113 08:56:20.343448 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:20.427978 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:20.582680 11607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:56:20.583174 11607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:56:20.583344 11607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:56:20.592387 11607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:56:20.592725 11607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.2
W20250113 08:56:20.595561 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:20.605401 11607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.2:43333
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.123.2
--webserver_port=39675
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.2
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:56:20.607743 11607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:56:20.610091 11607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:56:20.635332 11614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:20.635411 11613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:20.638337 11607 server_base.cc:1034] running on GCE node
W20250113 08:56:20.639509 11616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:20.640607 11607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:56:20.642241 11607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:56:20.646001 11607 hybrid_clock.cc:648] HybridClock initialized: now 1736758580645535 us; error 370 us; skew 500 ppm
I20250113 08:56:20.646656 11607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:56:20.650388 11607 webserver.cc:458] Webserver started at http://127.7.123.2:39675/ using document root <none> and password file <none>
I20250113 08:56:20.651901 11607 fs_manager.cc:362] Metadata directory not provided
I20250113 08:56:20.652244 11607 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:56:20.660526 11607 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250113 08:56:20.665489 11622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:20.667799 11607 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250113 08:56:20.668115 11607 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
uuid: "5f8ea89390544489a231c733997d2d9a"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:56:20.669284 11607 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250113 08:56:20.685499 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:56:20.700560 11607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:56:20.701761 11607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:56:20.702119 11607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:56:20.703910 11607 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:56:20.707069 11629 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:56:20.715310 11607 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:56:20.715512 11607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20250113 08:56:20.715656 11607 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:56:20.720086 11607 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:56:20.720305 11607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.001s sys 0.004s
I20250113 08:56:20.720885 11629 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap starting.
I20250113 08:56:20.784168 11607 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.2:43333
I20250113 08:56:20.784152 11735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.2:43333 every 8 connection(s)
I20250113 08:56:20.786163 11607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-1/data/info.pb
I20250113 08:56:20.787505 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 11607
I20250113 08:56:20.787884 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 10807
I20250113 08:56:20.830031 10932 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:56:20.830590 11736 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:56:20.831065 11736 heartbeater.cc:463] Registering TS with master...
I20250113 08:56:20.832402 11736 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:20.836382 8932 ts_manager.cc:194] Re-registered known tserver with Master: 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333)
I20250113 08:56:20.838481 8932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.2:60787
W20250113 08:56:20.861212 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:21.197348 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 10807
W20250113 08:56:21.197384 10851 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38078: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:21.263859 9534 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389) has failed: Network error: recv got EOF from 127.7.123.1:38389 (error 108)
I20250113 08:56:21.264341 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.1:38389
--local_ip_for_outbound_sockets=127.7.123.1
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=33665
--webserver_interface=127.7.123.1
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:56:21.318722 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.320168 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.345603 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.345619 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.385087 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.385950 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.432099 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.435685 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.447240 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.491827 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.493109 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.559298 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.560483 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.634902 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.635375 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.637717 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.719540 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.722076 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.810320 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.814337 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:21.838653 11743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:56:21.839215 11743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:56:21.839432 11743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250113 08:56:21.843761 11736 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
W20250113 08:56:21.855348 11743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:56:21.855754 11743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.1
I20250113 08:56:21.875217 11743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.1:38389
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.123.1
--webserver_port=33665
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.1
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:56:21.878213 11743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:56:21.881664 11743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250113 08:56:21.896939 11629 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Log is configured to *not* fsync() on all Append() calls
W20250113 08:56:21.903767 11753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:21.904803 11751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:21.905727 11750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:21.908267 11743 server_base.cc:1034] running on GCE node
I20250113 08:56:21.909350 11743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:56:21.911002 11743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20250113 08:56:21.912600 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:21.913257 11743 hybrid_clock.cc:648] HybridClock initialized: now 1736758581913083 us; error 150 us; skew 500 ppm
I20250113 08:56:21.913745 11743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250113 08:56:21.914778 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:21.917904 11743 webserver.cc:458] Webserver started at http://127.7.123.1:33665/ using document root <none> and password file <none>
I20250113 08:56:21.919401 11743 fs_manager.cc:362] Metadata directory not provided
I20250113 08:56:21.919598 11743 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250113 08:56:21.920210 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:21.928946 11743 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.003s sys 0.004s
I20250113 08:56:21.934193 11760 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:21.937207 11743 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20250113 08:56:21.937599 11743 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
uuid: "e3f389d6e3484a13af0780989a117f02"
format_stamp: "Formatted at 2025-01-13 08:55:04 on dist-test-slave-vxbw"
I20250113 08:56:21.938980 11743 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:56:21.983206 11743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:56:21.984628 11743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:56:21.985037 11743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:56:21.986495 11743 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:56:21.989316 11767 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:56:21.999833 11743 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:56:22.000028 11743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.001s
I20250113 08:56:22.000223 11743 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:56:22.004377 11743 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:56:22.004504 11743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.001s
I20250113 08:56:22.005057 11767 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap starting.
W20250113 08:56:22.020664 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.026160 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.185155 11124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.186583 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.218978 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:22.248952 11743 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.1:38389
I20250113 08:56:22.248849 11873 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.1:38389 every 8 connection(s)
I20250113 08:56:22.251345 11743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-0/data/info.pb
I20250113 08:56:22.259265 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 11743
I20250113 08:56:22.259590 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 10945
I20250113 08:56:22.317093 11067 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:56:22.345500 11874 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:56:22.346549 11874 heartbeater.cc:463] Registering TS with master...
I20250113 08:56:22.348846 11874 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:22.354085 8930 ts_manager.cc:194] Re-registered known tserver with Master: e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389)
I20250113 08:56:22.356164 8930 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.1:49349
W20250113 08:56:22.386583 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.424934 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:22.458351 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 10945
I20250113 08:56:22.485297 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.4:34591
--local_ip_for_outbound_sockets=127.7.123.4
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=45007
--webserver_interface=127.7.123.4
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:56:22.528723 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.572791 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.677832 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.681152 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.752610 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.844750 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:22.849617 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:23.013157 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:23.031242 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:23.086236 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:23.190281 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:23.195394 11882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:56:23.195958 11882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:56:23.196134 11882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:56:23.211483 11882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:56:23.211908 11882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.4
I20250113 08:56:23.230136 11882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.4:34591
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.7.123.4
--webserver_port=45007
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.4
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:56:23.232806 11882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:56:23.236696 11882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:56:23.266006 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:23.287914 11887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:23.291870 11888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:23.304409 11891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:23.306325 11882 server_base.cc:1034] running on GCE node
I20250113 08:56:23.307693 11882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20250113 08:56:23.309008 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:23.310535 11882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:56:23.312378 11882 hybrid_clock.cc:648] HybridClock initialized: now 1736758583312305 us; error 45 us; skew 500 ppm
I20250113 08:56:23.312912 11882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250113 08:56:23.320369 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:23.324232 11882 webserver.cc:458] Webserver started at http://127.7.123.4:45007/ using document root <none> and password file <none>
I20250113 08:56:23.325642 11882 fs_manager.cc:362] Metadata directory not provided
I20250113 08:56:23.325858 11882 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:56:23.336509 11882 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.007s sys 0.000s
I20250113 08:56:23.341487 11896 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:23.344141 11882 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250113 08:56:23.344399 11882 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
uuid: "00524bc1b1a34b61af38c9f2bded4031"
format_stamp: "Formatted at 2025-01-13 08:55:06 on dist-test-slave-vxbw"
I20250113 08:56:23.345880 11882 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:56:23.378260 11874 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:56:23.393970 11767 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Log is configured to *not* fsync() on all Append() calls
I20250113 08:56:23.410277 11882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:56:23.411675 11882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:56:23.412101 11882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:56:23.413674 11882 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:56:23.416285 11882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:56:23.416440 11882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:23.416703 11882 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:56:23.416818 11882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250113 08:56:23.454767 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:23.517522 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
W20250113 08:56:23.657259 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:23.694851 11882 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.4:34591
I20250113 08:56:23.694934 12009 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.4:34591 every 8 connection(s)
I20250113 08:56:23.697525 11882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-3/data/info.pb
I20250113 08:56:23.707337 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 11882
I20250113 08:56:23.707638 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 11080
W20250113 08:56:23.723482 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:23.747668 11204 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:56:23.759231 12010 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:56:23.759737 12010 heartbeater.cc:463] Registering TS with master...
I20250113 08:56:23.761209 12010 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:23.764510 8930 ts_manager.cc:194] Re-registered known tserver with Master: 00524bc1b1a34b61af38c9f2bded4031 (127.7.123.4:34591)
I20250113 08:56:23.766132 8930 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.4:50181
W20250113 08:56:24.230576 11123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58034: Illegal state: replica cf369533d8ad453f9d6c47cad3706dab is not leader of this config: current role FOLLOWER
I20250113 08:56:24.231639 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 11080
W20250113 08:56:24.329133 9534 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) has failed: Network error: recv got EOF from 127.7.123.3:43245 (error 108)
I20250113 08:56:24.329854 7660 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
/tmp/dist-test-taskmZWr4w/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.123.3:43245
--local_ip_for_outbound_sockets=127.7.123.3
--tserver_master_addrs=127.7.123.62:34471
--webserver_port=35523
--webserver_interface=127.7.123.3
--builtin_ntp_servers=127.7.123.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250113 08:56:24.771247 12010 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
W20250113 08:56:25.014675 12017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:56:25.015215 12017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:56:25.015637 12017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:56:25.030177 12017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:56:25.030586 12017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.123.3
I20250113 08:56:25.049556 12017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.123.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.123.3:43245
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.123.3
--webserver_port=35523
--tserver_master_addrs=127.7.123.62:34471
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.123.3
--log_dir=/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:56:25.052292 12017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:56:25.055747 12017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:56:25.082520 12023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:25.085906 12024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:25.095463 12017 server_base.cc:1034] running on GCE node
W20250113 08:56:25.096699 12026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:25.098156 12017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:56:25.103329 12017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:56:25.104974 12017 hybrid_clock.cc:648] HybridClock initialized: now 1736758585104871 us; error 93 us; skew 500 ppm
I20250113 08:56:25.105612 12017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:56:25.110553 12017 webserver.cc:458] Webserver started at http://127.7.123.3:35523/ using document root <none> and password file <none>
I20250113 08:56:25.112068 12017 fs_manager.cc:362] Metadata directory not provided
I20250113 08:56:25.112332 12017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:56:25.124521 12017 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.004s sys 0.002s
I20250113 08:56:25.133005 12032 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:25.135331 12017 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20250113 08:56:25.135635 12017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
uuid: "cf369533d8ad453f9d6c47cad3706dab"
format_stamp: "Formatted at 2025-01-13 08:55:05 on dist-test-slave-vxbw"
I20250113 08:56:25.137076 12017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:56:25.169230 12017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:56:25.172055 12017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:56:25.172477 12017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:56:25.175637 12017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:56:25.179242 12039 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:56:25.186264 12017 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:56:25.186450 12017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.002s sys 0.000s
I20250113 08:56:25.186720 12017 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:56:25.190244 12017 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:56:25.190407 12017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.001s sys 0.000s
I20250113 08:56:25.191130 12039 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap starting.
W20250113 08:56:25.416229 9534 meta_cache.cc:302] tablet b13658abfdf04a8c9fcc1f08c15ff92a: replica cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245) has failed: Network error: Client connection negotiation failed: client connection to 127.7.123.3:43245: connect: Connection refused (error 111)
I20250113 08:56:25.947558 12145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.123.3:43245 every 8 connection(s)
I20250113 08:56:25.947834 12017 rpc_server.cc:307] RPC server started. Bound to: 127.7.123.3:43245
I20250113 08:56:25.950376 12017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0/minicluster-data/ts-2/data/info.pb
I20250113 08:56:25.957865 7660 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu as pid 12017
I20250113 08:56:26.008507 12146 heartbeater.cc:346] Connected to a master server at 127.7.123.62:34471
I20250113 08:56:26.009063 12146 heartbeater.cc:463] Registering TS with master...
I20250113 08:56:26.010612 12146 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:26.014330 8930 ts_manager.cc:194] Re-registered known tserver with Master: cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:56:26.016260 8930 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.123.3:54283
I20250113 08:56:26.901682 11629 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap replaying log segment 1/1 (1.68M/5.31M this segment, stats: ops{read=975 overwritten=0 applied=974 ignored=0} inserts{seen=48600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250113 08:56:27.020740 12146 heartbeater.cc:502] Master 127.7.123.62:34471 was elected leader, sending a full tablet report...
I20250113 08:56:27.178218 11671 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:27.219949 12081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:27.250245 11945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:27.278635 11809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:27.288823 12039 log.cc:826] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Log is configured to *not* fsync() on all Append() calls
I20250113 08:56:28.397249 11767 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap replaying log segment 1/1 (1.92M/5.24M this segment, stats: ops{read=1109 overwritten=0 applied=1107 ignored=0} inserts{seen=55250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250113 08:56:31.901695 11629 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap replaying log segment 1/1 (3.41M/5.31M this segment, stats: ops{read=1971 overwritten=0 applied=1970 ignored=0} inserts{seen=98350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250113 08:56:32.304169 12039 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap replaying log segment 1/1 (1.83M/5.15M this segment, stats: ops{read=1062 overwritten=0 applied=1061 ignored=0} inserts{seen=52950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250113 08:56:33.400390 11767 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap replaying log segment 1/1 (4.11M/5.24M this segment, stats: ops{read=2398 overwritten=0 applied=2398 ignored=0} inserts{seen=119750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250113 08:56:33.877605 11945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:33.884402 11671 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:33.902886 12081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:33.924698 11809 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:36.097645 11767 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=3070 overwritten=0 applied=3068 ignored=0} inserts{seen=153200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 08:56:36.098760 11767 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Bootstrap complete.
I20250113 08:56:36.101013 11767 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Time spent bootstrapping tablet: real 14.096s user 12.567s sys 1.016s
I20250113 08:56:36.108109 11767 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:36.112365 11767 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f389d6e3484a13af0780989a117f02, State: Initialized, Role: FOLLOWER
I20250113 08:56:36.113508 11767 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3068, Last appended: 6.3070, Last appended by leader: 3070, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:36.116532 11767 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20250113 08:56:36.340876 12223 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:36.341192 12223 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:36.356633 12223 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:56:36.370266 12101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3070 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
W20250113 08:56:36.376816 11764 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Tablet error from VoteRequest() call to peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:36.374711 11691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3070 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a" is_pre_election: true
W20250113 08:56:36.379881 11763 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Tablet error from VoteRequest() call to peer 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:36.380098 11763 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3f389d6e3484a13af0780989a117f02; no voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab
I20250113 08:56:36.380722 12223 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250113 08:56:36.440388 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:36.633797 9534 batcher.cc:441] Timed out: Failed to write batch of 50 ops to tablet b13658abfdf04a8c9fcc1f08c15ff92a after 188 attempt(s): Failed to write to server: (no server available): Write(tablet: b13658abfdf04a8c9fcc1f08c15ff92a, num_ops: 50, num_attempts: 188) passed its deadline: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:36.650002 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:36.673621 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:36.710345 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:36.756960 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:56:36.810274 12223 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:36.810648 12223 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:36.812166 12223 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:56:36.813491 11691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3070 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a" is_pre_election: true
I20250113 08:56:36.814121 12101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3070 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
W20250113 08:56:36.814882 11764 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Tablet error from VoteRequest() call to peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Illegal state: must be running to vote when last-logged opid is not known
W20250113 08:56:36.815045 11763 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Tablet error from VoteRequest() call to peer 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:36.815191 11763 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3f389d6e3484a13af0780989a117f02; no voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab
I20250113 08:56:36.815647 12223 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250113 08:56:36.817037 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:36.874456 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:56:36.903344 11629 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap replaying log segment 1/1 (5.20M/5.31M this segment, stats: ops{read=3007 overwritten=0 applied=3007 ignored=0} inserts{seen=150200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:56:36.946208 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:37.019690 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:37.107261 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:56:37.141288 11629 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3071 overwritten=1 applied=3068 ignored=0} inserts{seen=153200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250113 08:56:37.142171 11629 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Bootstrap complete.
I20250113 08:56:37.144304 11629 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Time spent bootstrapping tablet: real 16.424s user 14.968s sys 1.029s
I20250113 08:56:37.149591 11629 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 7 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:37.155900 11629 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8ea89390544489a231c733997d2d9a, State: Initialized, Role: FOLLOWER
I20250113 08:56:37.157588 11629 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3068, Last appended: 6.3070, Last appended by leader: 3070, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:37.161207 11629 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
W20250113 08:56:37.207523 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
W20250113 08:56:37.249505 11650 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38522: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
W20250113 08:56:37.321480 11789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48622: Illegal state: replica e3f389d6e3484a13af0780989a117f02 is not leader of this config: current role FOLLOWER
I20250113 08:56:37.328282 12039 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap replaying log segment 1/1 (3.61M/5.15M this segment, stats: ops{read=2142 overwritten=0 applied=2103 ignored=0} inserts{seen=105000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:56:37.358767 11650 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38522: Illegal state: replica 5f8ea89390544489a231c733997d2d9a is not leader of this config: current role FOLLOWER
I20250113 08:56:37.397351 12230 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:56:37.397591 12230 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:37.398902 12230 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:56:37.399405 11691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3070 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a" is_pre_election: true
I20250113 08:56:37.399569 12101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3070 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
I20250113 08:56:37.399834 11691 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f389d6e3484a13af0780989a117f02 in term 7.
W20250113 08:56:37.400296 11764 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Tablet error from VoteRequest() call to peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:37.400552 11763 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a, e3f389d6e3484a13af0780989a117f02; no voters: cf369533d8ad453f9d6c47cad3706dab
I20250113 08:56:37.400971 12230 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250113 08:56:37.401240 12230 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:56:37.401477 12230 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:56:37.405721 12230 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:37.406612 12230 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 election: Requested vote from peers 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:56:37.407316 11691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3070 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a"
I20250113 08:56:37.408028 11691 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:56:37.408720 12101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "e3f389d6e3484a13af0780989a117f02" candidate_term: 8 candidate_status { last_received { term: 6 index: 3070 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab"
W20250113 08:56:37.409598 11764 leader_election.cc:343] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 election: Tablet error from VoteRequest() call to peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Illegal state: must be running to vote when last-logged opid is not known
I20250113 08:56:37.412894 11691 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3f389d6e3484a13af0780989a117f02 in term 8.
I20250113 08:56:37.413630 11763 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a, e3f389d6e3484a13af0780989a117f02; no voters: cf369533d8ad453f9d6c47cad3706dab
I20250113 08:56:37.414083 12230 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 8 FOLLOWER]: Leader election won for term 8
I20250113 08:56:37.414692 12230 raft_consensus.cc:695] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [term 8 LEADER]: Becoming Leader. State: Replica: e3f389d6e3484a13af0780989a117f02, State: Running, Role: LEADER
I20250113 08:56:37.415397 12230 consensus_queue.cc:237] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3068, Committed index: 3068, Last appended: 6.3070, Last appended by leader: 3070, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:37.421658 8932 catalog_manager.cc:5526] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "e3f389d6e3484a13af0780989a117f02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } health_report { overall_health: UNKNOWN } } }
I20250113 08:56:37.439829 11691 raft_consensus.cc:1270] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 8 FOLLOWER]: Refusing update from remote peer e3f389d6e3484a13af0780989a117f02: Log matching property violated. Preceding OpId in replica: term: 6 index: 3070. Preceding OpId from leader: term: 8 index: 3072. (index mismatch)
W20250113 08:56:37.440032 11764 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250113 08:56:37.440945 12230 consensus_queue.cc:1035] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3071, Last known committed idx: 3068, Time since last communication: 0.000s
I20250113 08:56:37.452255 12239 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48622 (ReqId={client: 174b6e3e9f79481394e37ad662f9ad1a, seq_no=3064, attempt_no=164}) took 1182 ms. Trace:
I20250113 08:56:37.452608 12239 rpcz_store.cc:276] 0113 08:56:36.270118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:56:36.270267 (+ 149us) service_pool.cc:225] Handling call
0113 08:56:37.452173 (+1181906us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250113 08:56:37.457275 12237 mvcc.cc:204] Tried to move back new op lower bound from 7113763215101997056 to 7113763215019393024. Current Snapshot: MvccSnapshot[applied={T|T < 7113763215101997056 or (T in {7113763215101997056})}]
W20250113 08:56:37.975183 11764 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250113 08:56:38.499213 11764 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250113 08:56:38.988045 11764 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
W20250113 08:56:39.518231 11764 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
W20250113 08:56:40.007265 11764 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250113 08:56:40.420176 11945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:40.434594 12081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:40.477283 11671 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:56:40.538157 11764 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250113 08:56:40.542625 11809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20250113 08:56:41.075279 11764 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
W20250113 08:56:41.122223 9534 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.7.123.1:38389, user_credentials={real_user=slave}} blocked reactor thread for 63578.7us
W20250113 08:56:41.621467 11764 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
I20250113 08:56:41.981696 12244 consensus_queue.cc:786] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Peer cf369533d8ad453f9d6c47cad3706dab is lagging by at least 7 ops behind the committed index
W20250113 08:56:42.169301 11764 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 -> Peer cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245): Couldn't send request to peer cf369533d8ad453f9d6c47cad3706dab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
I20250113 08:56:42.299055 12039 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap replayed 1/1 log segments. Stats: ops{read=3068 overwritten=0 applied=3067 ignored=0} inserts{seen=153150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250113 08:56:42.300038 12039 tablet_bootstrap.cc:492] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Bootstrap complete.
I20250113 08:56:42.303031 12039 ts_tablet_manager.cc:1397] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Time spent bootstrapping tablet: real 17.112s user 15.228s sys 1.170s
I20250113 08:56:42.310062 12039 raft_consensus.cc:357] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 7 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:42.316041 12039 raft_consensus.cc:738] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf369533d8ad453f9d6c47cad3706dab, State: Initialized, Role: FOLLOWER
I20250113 08:56:42.317330 12039 consensus_queue.cc:260] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3067, Last appended: 6.3068, Last appended by leader: 3068, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:42.321556 12039 ts_tablet_manager.cc:1428] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab: Time spent starting tablet: real 0.018s user 0.014s sys 0.002s
I20250113 08:56:42.343657 12101 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 7 FOLLOWER]: Advancing to term 8
I20250113 08:56:42.350978 12101 raft_consensus.cc:1270] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 8 FOLLOWER]: Refusing update from remote peer e3f389d6e3484a13af0780989a117f02: Log matching property violated. Preceding OpId in replica: term: 6 index: 3068. Preceding OpId from leader: term: 6 index: 3070. (index mismatch)
I20250113 08:56:42.360352 12245 consensus_queue.cc:1037] T b13658abfdf04a8c9fcc1f08c15ff92a P e3f389d6e3484a13af0780989a117f02 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3071, Last known committed idx: 3067, Time since last communication: 0.000s
I20250113 08:56:43.219717 12288 mvcc.cc:204] Tried to move back new op lower bound from 7113763233278259200 to 7113763215019393024. Current Snapshot: MvccSnapshot[applied={T|T < 7113763233278259200}]
I20250113 08:56:47.223589 11945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:47.244097 11671 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:47.308027 12081 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:47.340276 11809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:56:49.063854 8930 ts_manager.cc:281] Unset tserver state for 5f8ea89390544489a231c733997d2d9a from MAINTENANCE_MODE
I20250113 08:56:49.066063 11874 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:49.066612 12010 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:49.067065 12146 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:49.082129 8937 ts_manager.cc:281] Unset tserver state for 00524bc1b1a34b61af38c9f2bded4031 from MAINTENANCE_MODE
I20250113 08:56:49.240108 8933 ts_manager.cc:281] Unset tserver state for cf369533d8ad453f9d6c47cad3706dab from MAINTENANCE_MODE
I20250113 08:56:49.242755 8937 ts_manager.cc:281] Unset tserver state for e3f389d6e3484a13af0780989a117f02 from MAINTENANCE_MODE
I20250113 08:56:49.522469 11736 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:50.083043 11874 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:50.091511 12010 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
I20250113 08:56:50.096679 12146 heartbeater.cc:510] Master 127.7.123.62:34471 requested a full tablet report, sending...
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/maintenance_mode-itest.cc:750: Failure
Value of: s.ok()
Actual: true
Expected: false
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:401: Failure
Failed
Timed out waiting for assertion to pass.
I20250113 08:56:52.169595 7660 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250113 08:56:52.169868 7660 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID e3f389d6e3484a13af0780989a117f02 and pid 11743
************************ BEGIN STACKS **************************
[New LWP 11746]
[New LWP 11747]
[New LWP 11748]
[New LWP 11749]
[New LWP 11756]
[New LWP 11757]
[New LWP 11758]
[New LWP 11761]
[New LWP 11762]
[New LWP 11763]
[New LWP 11764]
[New LWP 11765]
[New LWP 11766]
[New LWP 11768]
[New LWP 11769]
[New LWP 11770]
[New LWP 11771]
[New LWP 11772]
[New LWP 11773]
[New LWP 11774]
[New LWP 11775]
[New LWP 11776]
[New LWP 11777]
[New LWP 11778]
[New LWP 11779]
[New LWP 11780]
[New LWP 11781]
[New LWP 11782]
[New LWP 11783]
[New LWP 11784]
[New LWP 11785]
[New LWP 11786]
[New LWP 11787]
[New LWP 11788]
[New LWP 11789]
[New LWP 11790]
[New LWP 11791]
[New LWP 11792]
[New LWP 11793]
[New LWP 11794]
[New LWP 11795]
[New LWP 11796]
[New LWP 11797]
[New LWP 11798]
[New LWP 11799]
[New LWP 11800]
[New LWP 11801]
[New LWP 11802]
[New LWP 11803]
[New LWP 11804]
[New LWP 11805]
[New LWP 11806]
[New LWP 11807]
[New LWP 11808]
[New LWP 11809]
[New LWP 11810]
[New LWP 11811]
[New LWP 11812]
[New LWP 11813]
[New LWP 11814]
[New LWP 11815]
[New LWP 11816]
[New LWP 11817]
[New LWP 11818]
[New LWP 11819]
[New LWP 11820]
[New LWP 11821]
[New LWP 11822]
[New LWP 11823]
[New LWP 11824]
[New LWP 11825]
[New LWP 11826]
[New LWP 11827]
[New LWP 11828]
[New LWP 11829]
[New LWP 11830]
[New LWP 11831]
[New LWP 11832]
[New LWP 11833]
[New LWP 11834]
[New LWP 11835]
[New LWP 11836]
[New LWP 11837]
[New LWP 11838]
[New LWP 11839]
[New LWP 11840]
[New LWP 11841]
[New LWP 11842]
[New LWP 11843]
[New LWP 11844]
[New LWP 11845]
[New LWP 11846]
[New LWP 11847]
[New LWP 11848]
[New LWP 11849]
[New LWP 11850]
[New LWP 11851]
[New LWP 11852]
[New LWP 11853]
[New LWP 11854]
[New LWP 11855]
[New LWP 11856]
[New LWP 11857]
[New LWP 11858]
[New LWP 11859]
[New LWP 11860]
[New LWP 11861]
[New LWP 11862]
[New LWP 11863]
[New LWP 11864]
[New LWP 11865]
[New LWP 11866]
[New LWP 11867]
[New LWP 11868]
[New LWP 11869]
[New LWP 11870]
[New LWP 11871]
[New LWP 11872]
[New LWP 11873]
[New LWP 11874]
[New LWP 11875]
[New LWP 12232]
[New LWP 12371]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f4f8a6ebd50 in ?? ()
Id Target Id Frame
* 1 LWP 11743 "kudu" 0x00007f4f8a6ebd50 in ?? ()
2 LWP 11746 "kudu" 0x00007f4f8a6e7fb9 in ?? ()
3 LWP 11747 "kudu" 0x00007f4f8a6e7fb9 in ?? ()
4 LWP 11748 "kudu" 0x00007f4f8a6e7fb9 in ?? ()
5 LWP 11749 "kernel-watcher-" 0x00007f4f8a6e7fb9 in ?? ()
6 LWP 11756 "ntp client-1175" 0x00007f4f8a6eb9e2 in ?? ()
7 LWP 11757 "file cache-evic" 0x00007f4f8a6e7fb9 in ?? ()
8 LWP 11758 "sq_acceptor" 0x00007f4f83444cb9 in ?? ()
9 LWP 11761 "rpc reactor-117" 0x00007f4f83451a47 in ?? ()
10 LWP 11762 "rpc reactor-117" 0x00007f4f83451a47 in ?? ()
11 LWP 11763 "rpc reactor-117" 0x00007f4f83451a47 in ?? ()
12 LWP 11764 "rpc reactor-117" 0x00007f4f83451a47 in ?? ()
13 LWP 11765 "MaintenanceMgr " 0x00007f4f8a6e7ad3 in ?? ()
14 LWP 11766 "txn-status-mana" 0x00007f4f8a6e7fb9 in ?? ()
15 LWP 11768 "collect_and_rem" 0x00007f4f8a6e7fb9 in ?? ()
16 LWP 11769 "tc-session-exp-" 0x00007f4f8a6e7fb9 in ?? ()
17 LWP 11770 "rpc worker-1177" 0x00007f4f8a6e7ad3 in ?? ()
18 LWP 11771 "rpc worker-1177" 0x00007f4f8a6e7ad3 in ?? ()
19 LWP 11772 "rpc worker-1177" 0x00007f4f8a6e7ad3 in ?? ()
20 LWP 11773 "rpc worker-1177" 0x00007f4f8a6e7ad3 in ?? ()
21 LWP 11774 "rpc worker-1177" 0x00007f4f8a6e7ad3 in ?? ()
22 LWP 11775 "rpc worker-1177" 0x00007f4f8a6e7ad3 in ?? ()
23 LWP 11776 "rpc worker-1177" 0x00007f4f8a6e7ad3 in ?? ()
24 LWP 11777 "rpc worker-1177" 0x00007f4f8a6e7ad3 in ?? ()
25 LWP 11778 "rpc worker-1177" 0x00007f4f8a6e7ad3 in ?? ()
26 LWP 11779 "rpc worker-1177" 0x00007f4f8a6e7ad3 in ?? ()
27 LWP 11780 "rpc worker-1178" 0x00007f4f8a6e7ad3 in ?? ()
28 LWP 11781 "rpc worker-1178" 0x00007f4f8a6e7ad3 in ?? ()
29 LWP 11782 "rpc worker-1178" 0x00007f4f8a6e7ad3 in ?? ()
30 LWP 11783 "rpc worker-1178" 0x00007f4f8a6e7ad3 in ?? ()
31 LWP 11784 "rpc worker-1178" 0x00007f4f8a6e7ad3 in ?? ()
32 LWP 11785 "rpc worker-1178" 0x00007f4f8a6e7ad3 in ?? ()
33 LWP 11786 "rpc worker-1178" 0x00007f4f8a6e7ad3 in ?? ()
34 LWP 11787 "rpc worker-1178" 0x00007f4f8a6e7ad3 in ?? ()
35 LWP 11788 "rpc worker-1178" 0x00007f4f8a6e7ad3 in ?? ()
36 LWP 11789 "rpc worker-1178" 0x00007f4f8a6e7ad3 in ?? ()
37 LWP 11790 "rpc worker-1179" 0x00007f4f8a6e7ad3 in ?? ()
38 LWP 11791 "rpc worker-1179" 0x00007f4f8a6e7ad3 in ?? ()
39 LWP 11792 "rpc worker-1179" 0x00007f4f8a6e7ad3 in ?? ()
40 LWP 11793 "rpc worker-1179" 0x00007f4f8a6e7ad3 in ?? ()
41 LWP 11794 "rpc worker-1179" 0x00007f4f8a6e7ad3 in ?? ()
42 LWP 11795 "rpc worker-1179" 0x00007f4f8a6e7ad3 in ?? ()
43 LWP 11796 "rpc worker-1179" 0x00007f4f8a6e7ad3 in ?? ()
44 LWP 11797 "rpc worker-1179" 0x00007f4f8a6e7ad3 in ?? ()
45 LWP 11798 "rpc worker-1179" 0x00007f4f8a6e7ad3 in ?? ()
46 LWP 11799 "rpc worker-1179" 0x00007f4f8a6e7ad3 in ?? ()
47 LWP 11800 "rpc worker-1180" 0x00007f4f8a6e7ad3 in ?? ()
48 LWP 11801 "rpc worker-1180" 0x00007f4f8a6e7ad3 in ?? ()
49 LWP 11802 "rpc worker-1180" 0x00007f4f8a6e7ad3 in ?? ()
50 LWP 11803 "rpc worker-1180" 0x00007f4f8a6e7ad3 in ?? ()
51 LWP 11804 "rpc worker-1180" 0x00007f4f8a6e7ad3 in ?? ()
52 LWP 11805 "rpc worker-1180" 0x00007f4f8a6e7ad3 in ?? ()
53 LWP 11806 "rpc worker-1180" 0x00007f4f8a6e7ad3 in ?? ()
54 LWP 11807 "rpc worker-1180" 0x00007f4f8a6e7ad3 in ?? ()
55 LWP 11808 "rpc worker-1180" 0x00007f4f8a6e7ad3 in ?? ()
56 LWP 11809 "rpc worker-1180" 0x00007f4f8a6e7ad3 in ?? ()
57 LWP 11810 "rpc worker-1181" 0x00007f4f8a6e7ad3 in ?? ()
58 LWP 11811 "rpc worker-1181" 0x00007f4f8a6e7ad3 in ?? ()
59 LWP 11812 "rpc worker-1181" 0x00007f4f8a6e7ad3 in ?? ()
60 LWP 11813 "rpc worker-1181" 0x00007f4f8a6e7ad3 in ?? ()
61 LWP 11814 "rpc worker-1181" 0x00007f4f8a6e7ad3 in ?? ()
62 LWP 11815 "rpc worker-1181" 0x00007f4f8a6e7ad3 in ?? ()
63 LWP 11816 "rpc worker-1181" 0x00007f4f8a6e7ad3 in ?? ()
64 LWP 11817 "rpc worker-1181" 0x00007f4f8a6e7ad3 in ?? ()
65 LWP 11818 "rpc worker-1181" 0x00007f4f8a6e7ad3 in ?? ()
66 LWP 11819 "rpc worker-1181" 0x00007f4f8a6e7ad3 in ?? ()
67 LWP 11820 "rpc worker-1182" 0x00007f4f8a6e7ad3 in ?? ()
68 LWP 11821 "rpc worker-1182" 0x00007f4f8a6e7ad3 in ?? ()
69 LWP 11822 "rpc worker-1182" 0x00007f4f8a6e7ad3 in ?? ()
70 LWP 11823 "rpc worker-1182" 0x00007f4f8a6e7ad3 in ?? ()
71 LWP 11824 "rpc worker-1182" 0x00007f4f8a6e7ad3 in ?? ()
72 LWP 11825 "rpc worker-1182" 0x00007f4f8a6e7ad3 in ?? ()
73 LWP 11826 "rpc worker-1182" 0x00007f4f8a6e7ad3 in ?? ()
74 LWP 11827 "rpc worker-1182" 0x00007f4f8a6e7ad3 in ?? ()
75 LWP 11828 "rpc worker-1182" 0x00007f4f8a6e7ad3 in ?? ()
76 LWP 11829 "rpc worker-1182" 0x00007f4f8a6e7ad3 in ?? ()
77 LWP 11830 "rpc worker-1183" 0x00007f4f8a6e7ad3 in ?? ()
78 LWP 11831 "rpc worker-1183" 0x00007f4f8a6e7ad3 in ?? ()
79 LWP 11832 "rpc worker-1183" 0x00007f4f8a6e7ad3 in ?? ()
80 LWP 11833 "rpc worker-1183" 0x00007f4f8a6e7ad3 in ?? ()
81 LWP 11834 "rpc worker-1183" 0x00007f4f8a6e7ad3 in ?? ()
82 LWP 11835 "rpc worker-1183" 0x00007f4f8a6e7ad3 in ?? ()
83 LWP 11836 "rpc worker-1183" 0x00007f4f8a6e7ad3 in ?? ()
84 LWP 11837 "rpc worker-1183" 0x00007f4f8a6e7ad3 in ?? ()
85 LWP 11838 "rpc worker-1183" 0x00007f4f8a6e7ad3 in ?? ()
86 LWP 11839 "rpc worker-1183" 0x00007f4f8a6e7ad3 in ?? ()
87 LWP 11840 "rpc worker-1184" 0x00007f4f8a6e7ad3 in ?? ()
88 LWP 11841 "rpc worker-1184" 0x00007f4f8a6e7ad3 in ?? ()
89 LWP 11842 "rpc worker-1184" 0x00007f4f8a6e7ad3 in ?? ()
90 LWP 11843 "rpc worker-1184" 0x00007f4f8a6e7ad3 in ?? ()
91 LWP 11844 "rpc worker-1184" 0x00007f4f8a6e7ad3 in ?? ()
92 LWP 11845 "rpc worker-1184" 0x00007f4f8a6e7ad3 in ?? ()
93 LWP 11846 "rpc worker-1184" 0x00007f4f8a6e7ad3 in ?? ()
94 LWP 11847 "rpc worker-1184" 0x00007f4f8a6e7ad3 in ?? ()
95 LWP 11848 "rpc worker-1184" 0x00007f4f8a6e7ad3 in ?? ()
96 LWP 11849 "rpc worker-1184" 0x00007f4f8a6e7ad3 in ?? ()
97 LWP 11850 "rpc worker-1185" 0x00007f4f8a6e7ad3 in ?? ()
98 LWP 11851 "rpc worker-1185" 0x00007f4f8a6e7ad3 in ?? ()
99 LWP 11852 "rpc worker-1185" 0x00007f4f8a6e7ad3 in ?? ()
100 LWP 11853 "rpc worker-1185" 0x00007f4f8a6e7ad3 in ?? ()
101 LWP 11854 "rpc worker-1185" 0x00007f4f8a6e7ad3 in ?? ()
102 LWP 11855 "rpc worker-1185" 0x00007f4f8a6e7ad3 in ?? ()
103 LWP 11856 "rpc worker-1185" 0x00007f4f8a6e7ad3 in ?? ()
104 LWP 11857 "rpc worker-1185" 0x00007f4f8a6e7ad3 in ?? ()
105 LWP 11858 "rpc worker-1185" 0x00007f4f8a6e7ad3 in ?? ()
106 LWP 11859 "rpc worker-1185" 0x00007f4f8a6e7ad3 in ?? ()
107 LWP 11860 "rpc worker-1186" 0x00007f4f8a6e7ad3 in ?? ()
108 LWP 11861 "rpc worker-1186" 0x00007f4f8a6e7ad3 in ?? ()
109 LWP 11862 "rpc worker-1186" 0x00007f4f8a6e7ad3 in ?? ()
110 LWP 11863 "rpc worker-1186" 0x00007f4f8a6e7ad3 in ?? ()
111 LWP 11864 "rpc worker-1186" 0x00007f4f8a6e7ad3 in ?? ()
112 LWP 11865 "rpc worker-1186" 0x00007f4f8a6e7ad3 in ?? ()
113 LWP 11866 "rpc worker-1186" 0x00007f4f8a6e7ad3 in ?? ()
114 LWP 11867 "rpc worker-1186" 0x00007f4f8a6e7ad3 in ?? ()
115 LWP 11868 "rpc worker-1186" 0x00007f4f8a6e7ad3 in ?? ()
116 LWP 11869 "rpc worker-1186" 0x00007f4f8a6e7ad3 in ?? ()
117 LWP 11870 "diag-logger-118" 0x00007f4f8a6e7fb9 in ?? ()
118 LWP 11871 "result-tracker-" 0x00007f4f8a6e7fb9 in ?? ()
119 LWP 11872 "excess-log-dele" 0x00007f4f8a6e7fb9 in ?? ()
120 LWP 11873 "acceptor-11873" 0x00007f4f834530c7 in ?? ()
121 LWP 11874 "heartbeat-11874" 0x00007f4f8a6e7fb9 in ?? ()
122 LWP 11875 "maintenance_sch" 0x00007f4f8a6e7fb9 in ?? ()
123 LWP 12232 "wal-append [wor" 0x00007f4f8a6e7fb9 in ?? ()
124 LWP 12371 "raft [worker]-1" 0x00007f4f8a6e7fb9 in ?? ()
Thread 124 (LWP 12371):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000123 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007f4f7c4b7b24 in ?? ()
#5 0x00007f4f7c4b77b0 in ?? ()
#6 0x0000000000000247 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f4f7c4b77d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 123 (LWP 12232):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x00006020003dfd78 in ?? ()
#2 0x00000000000008dc in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061a0004d05b0 in ?? ()
#5 0x00007f4f3c362090 in ?? ()
#6 0x00000000000011b8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 11875):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000007a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f4f3ef5b550 in ?? ()
#6 0x00000000000000f4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 11874):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0xcf5ed54e759050c7 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007f4f3f7735b0 in ?? ()
#6 0x0000000000000040 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 11873):
#0 0x00007f4f834530c7 in ?? ()
#1 0x0000614000021aa8 in ?? ()
#2 0x00007f4f3ff9ad50 in ?? ()
#3 0x00007f4f3ff9ad80 in ?? ()
#4 0x00007f4f3ff9ae80 in ?? ()
#5 0x00007f4f3ff9ad70 in ?? ()
#6 0x00007f4f3ff9ade0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007f4f3ff9af00 in ?? ()
#11 0x00007f4f3ff9a63c in ?? ()
#12 0x000000323ff9a5b0 in ?? ()
#13 0x00007f4f3f79e000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 11872):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x00007f4f407b3f00 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 11871):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x00007f4f40fcd0c0 in ?? ()
#2 0x000000000000001e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f4f40fcd0b0 in ?? ()
#6 0x000000000000003c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 11870):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 11869):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002c6838 in ?? ()
#4 0x00007f4f41ffce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f41ffce70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 11868):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 11867):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 11866):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 11865):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 11864):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 11863):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 11862):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 11861):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 11860):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 11859):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 11858):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 11857):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 11856):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 11855):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 11854):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 11853):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 11852):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 11851):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 11850):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 11849):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 11848):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 11847):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 11846):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 11845):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 11844):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 11843):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 11842):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 11841):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 11840):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 11839):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 11838):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 11837):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 11836):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 11835):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 11834):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 11833):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 11832):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 11831):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 11830):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 11829):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001c0068 in ?? ()
#4 0x00007f4f563cde50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f563cde70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 11828):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 11827):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 11826):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 11825):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 11824):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 11823):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 11822):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 11821):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 11820):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 11819):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 11818):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 11817):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 11816):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 11815):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 11814):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 11813):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 11812):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 11811):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 11810):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 11809):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000136828 in ?? ()
#4 0x00007f4f605aee50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f605aee70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 11808):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 11807):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 11806):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 11805):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 11804):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 11803):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 11802):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 11801):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 11800):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 11799):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 11798):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 11797):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 11796):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 11795):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 11794):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 11793):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 11792):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 11791):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 11790):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 11789):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x00000000000001e1 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000acfec in ?? ()
#4 0x00007f4f6a78fe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f6a78fe70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000acfa0 in ?? ()
#9 0x00007f4f8a6e7770 in ?? ()
#10 0x00007f4f6a78fe70 in ?? ()
#11 0x00000fe9ed4f1fd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 11788):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x000000000000026c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007f4f6afa7e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f6afa7e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 11787):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000106 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007f4f6b7bfe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f6b7bfe70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 11786):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 11785):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 11784):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 11783):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 11782):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 11781):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 11780):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 11779):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 11778):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 11777):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 11776):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 11775):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 11774):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 11773):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 11772):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 11771):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 11770):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 11769):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x00007f4f749a4c80 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007f4f749a4c70 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 11768):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 11766):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x00007f4f762291c0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007f4f762291b0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 11765):
#0 0x00007f4f8a6e7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 11764):
#0 0x00007f4f83451a47 in ?? ()
#1 0x00007f4f772782e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f4f772782d0 in ?? ()
#4 0x00007f4f772784e0 in ?? ()
#5 0x00007f4f77278320 in ?? ()
#6 0x0000614000021e98 in ?? ()
#7 0x00007f4f772783a0 in ?? ()
#8 0x00007f4f85ab125d in ?? ()
#9 0x3f9b8c038ca40000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000019000000 in ?? ()
#14 0x00007f4f00000000 in ?? ()
#15 0x41d9e134dfe3c367 in ?? ()
#16 0x00000fea6ee47074 in ?? ()
#17 0x00007f4f77278380 in ?? ()
#18 0x00007f4f85ab5ba3 in ?? ()
#19 0x00007f4f77278350 in ?? ()
#20 0x3f9b8c038ca40000 in ?? ()
#21 0x00000000772783a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021e98 in ?? ()
#24 0x3f9b8c038ca40000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 11763):
#0 0x00007f4f83451a47 in ?? ()
#1 0x00007f4f77a8f2e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f4f77a8f2d0 in ?? ()
#4 0x00007f4f77a8f4e0 in ?? ()
#5 0x00007f4f77a8f320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007f4f77a8f3a0 in ?? ()
#8 0x00007f4f85ab125d in ?? ()
#9 0x3f99bc1b79cd0000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x000000005a100000 in ?? ()
#14 0x00007f4f00000000 in ?? ()
#15 0x41d9e134dfe3c367 in ?? ()
#16 0x00000fea6ef49e74 in ?? ()
#17 0x00007f4f77a8f380 in ?? ()
#18 0x00007f4f85ab5ba3 in ?? ()
#19 0x00007f4f77a8f350 in ?? ()
#20 0x3f99bc1b79cd0000 in ?? ()
#21 0x0000000077a8f3a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3f99bc1b79cd0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 11762):
#0 0x00007f4f83451a47 in ?? ()
#1 0x00007f4f782a62e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f4f782a62d0 in ?? ()
#4 0x00007f4f782a64e0 in ?? ()
#5 0x00007f4f782a6320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007f4f782a63a0 in ?? ()
#8 0x00007f4f85ab125d in ?? ()
#9 0x3f9ea20a34330000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x00000000a7000000 in ?? ()
#14 0x00007f4f00000000 in ?? ()
#15 0x41d9e134dfe3c36a in ?? ()
#16 0x00000fea6f04cc74 in ?? ()
#17 0x00007f4f782a6380 in ?? ()
#18 0x00007f4f85ab5ba3 in ?? ()
#19 0x00007f4f782a6350 in ?? ()
#20 0x3f9ea20a34330000 in ?? ()
#21 0x00000000782a63a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3f9ea20a34330000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 11761):
#0 0x00007f4f83451a47 in ?? ()
#1 0x00007f4f7a2ad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f4f7a2ad2d0 in ?? ()
#4 0x00007f4f7a2ad4e0 in ?? ()
#5 0x00007f4f7a2ad320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f4f7a2ad3a0 in ?? ()
#8 0x00007f4f85ab125d in ?? ()
#9 0x3fb96ae86bc17000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000f6400000 in ?? ()
#14 0x00007f4f00000000 in ?? ()
#15 0x41d9e134dfe3c367 in ?? ()
#16 0x00000fea6f44da74 in ?? ()
#17 0x00007f4f7a2ad380 in ?? ()
#18 0x00007f4f85ab5ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 11758):
#0 0x00007f4f83444cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f4f7bcb7788 in ?? ()
#3 0x0000602000019790 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 11757):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x00007f4f7b4b6080 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 11756):
#0 0x00007f4f8a6eb9e2 in ?? ()
#1 0x00007f4f7acb3b60 in ?? ()
#2 0x00007f4f7acb3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f4f7acb3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f4f7acb3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f4f7acb3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f4f9000ff0c in ?? ()
#12 0x00007f4f8ffff7b9 in ?? ()
#13 0x00007f4f90003da6 in ?? ()
#14 0x00007f4f900087e0 in ?? ()
#15 0x00007f4f90008609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f4f872fe0be in ?? ()
#18 0x00007f4f8a6e16db in ?? ()
#19 0x00000fea6f58e67c in ?? ()
#20 0x00007f4f7acb3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 11749):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x00007f4f7ccb8c40 in ?? ()
#2 0x000000000000009a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f4f7ccb8c30 in ?? ()
#6 0x0000000000000134 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 11748):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 11747):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 11746):
#0 0x00007f4f8a6e7fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f4f7e4bc410 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 11743):
#0 0x00007f4f8a6ebd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 08:56:53.011482 7660 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID 5f8ea89390544489a231c733997d2d9a and pid 11607
************************ BEGIN STACKS **************************
[New LWP 11609]
[New LWP 11610]
[New LWP 11611]
[New LWP 11612]
[New LWP 11618]
[New LWP 11619]
[New LWP 11620]
[New LWP 11623]
[New LWP 11624]
[New LWP 11625]
[New LWP 11626]
[New LWP 11627]
[New LWP 11628]
[New LWP 11630]
[New LWP 11631]
[New LWP 11632]
[New LWP 11633]
[New LWP 11634]
[New LWP 11635]
[New LWP 11636]
[New LWP 11637]
[New LWP 11638]
[New LWP 11639]
[New LWP 11640]
[New LWP 11641]
[New LWP 11642]
[New LWP 11643]
[New LWP 11644]
[New LWP 11645]
[New LWP 11646]
[New LWP 11647]
[New LWP 11648]
[New LWP 11649]
[New LWP 11650]
[New LWP 11651]
[New LWP 11652]
[New LWP 11653]
[New LWP 11654]
[New LWP 11655]
[New LWP 11656]
[New LWP 11657]
[New LWP 11658]
[New LWP 11659]
[New LWP 11660]
[New LWP 11661]
[New LWP 11662]
[New LWP 11663]
[New LWP 11664]
[New LWP 11665]
[New LWP 11666]
[New LWP 11667]
[New LWP 11668]
[New LWP 11669]
[New LWP 11670]
[New LWP 11671]
[New LWP 11672]
[New LWP 11673]
[New LWP 11674]
[New LWP 11675]
[New LWP 11676]
[New LWP 11677]
[New LWP 11678]
[New LWP 11679]
[New LWP 11680]
[New LWP 11681]
[New LWP 11682]
[New LWP 11683]
[New LWP 11684]
[New LWP 11685]
[New LWP 11686]
[New LWP 11687]
[New LWP 11688]
[New LWP 11689]
[New LWP 11690]
[New LWP 11691]
[New LWP 11692]
[New LWP 11693]
[New LWP 11694]
[New LWP 11695]
[New LWP 11696]
[New LWP 11697]
[New LWP 11698]
[New LWP 11699]
[New LWP 11700]
[New LWP 11701]
[New LWP 11702]
[New LWP 11703]
[New LWP 11704]
[New LWP 11705]
[New LWP 11706]
[New LWP 11707]
[New LWP 11708]
[New LWP 11709]
[New LWP 11710]
[New LWP 11711]
[New LWP 11712]
[New LWP 11713]
[New LWP 11714]
[New LWP 11715]
[New LWP 11716]
[New LWP 11717]
[New LWP 11718]
[New LWP 11719]
[New LWP 11720]
[New LWP 11721]
[New LWP 11722]
[New LWP 11723]
[New LWP 11724]
[New LWP 11725]
[New LWP 11726]
[New LWP 11727]
[New LWP 11728]
[New LWP 11729]
[New LWP 11730]
[New LWP 11731]
[New LWP 11732]
[New LWP 11733]
[New LWP 11734]
[New LWP 11735]
[New LWP 11736]
[New LWP 11737]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f6217db1d50 in ?? ()
Id Target Id Frame
* 1 LWP 11607 "kudu" 0x00007f6217db1d50 in ?? ()
2 LWP 11609 "kudu" 0x00007f6217dadfb9 in ?? ()
3 LWP 11610 "kudu" 0x00007f6217dadfb9 in ?? ()
4 LWP 11611 "kudu" 0x00007f6217dadfb9 in ?? ()
5 LWP 11612 "kernel-watcher-" 0x00007f6217dadfb9 in ?? ()
6 LWP 11618 "ntp client-1161" 0x00007f6217db19e2 in ?? ()
7 LWP 11619 "file cache-evic" 0x00007f6217dadfb9 in ?? ()
8 LWP 11620 "sq_acceptor" 0x00007f6210b0acb9 in ?? ()
9 LWP 11623 "rpc reactor-116" 0x00007f6210b17a47 in ?? ()
10 LWP 11624 "rpc reactor-116" 0x00007f6210b17a47 in ?? ()
11 LWP 11625 "rpc reactor-116" 0x00007f6210b17a47 in ?? ()
12 LWP 11626 "rpc reactor-116" 0x00007f6210b17a47 in ?? ()
13 LWP 11627 "MaintenanceMgr " 0x00007f6217dadad3 in ?? ()
14 LWP 11628 "txn-status-mana" 0x00007f6217dadfb9 in ?? ()
15 LWP 11630 "collect_and_rem" 0x00007f6217dadfb9 in ?? ()
16 LWP 11631 "tc-session-exp-" 0x00007f6217dadfb9 in ?? ()
17 LWP 11632 "rpc worker-1163" 0x00007f6217dadad3 in ?? ()
18 LWP 11633 "rpc worker-1163" 0x00007f6217dadad3 in ?? ()
19 LWP 11634 "rpc worker-1163" 0x00007f6217dadad3 in ?? ()
20 LWP 11635 "rpc worker-1163" 0x00007f6217dadad3 in ?? ()
21 LWP 11636 "rpc worker-1163" 0x00007f6217dadad3 in ?? ()
22 LWP 11637 "rpc worker-1163" 0x00007f6217dadad3 in ?? ()
23 LWP 11638 "rpc worker-1163" 0x00007f6217dadad3 in ?? ()
24 LWP 11639 "rpc worker-1163" 0x00007f6217dadad3 in ?? ()
25 LWP 11640 "rpc worker-1164" 0x00007f6217dadad3 in ?? ()
26 LWP 11641 "rpc worker-1164" 0x00007f6217dadad3 in ?? ()
27 LWP 11642 "rpc worker-1164" 0x00007f6217dadad3 in ?? ()
28 LWP 11643 "rpc worker-1164" 0x00007f6217dadad3 in ?? ()
29 LWP 11644 "rpc worker-1164" 0x00007f6217dadad3 in ?? ()
30 LWP 11645 "rpc worker-1164" 0x00007f6217dadad3 in ?? ()
31 LWP 11646 "rpc worker-1164" 0x00007f6217dadad3 in ?? ()
32 LWP 11647 "rpc worker-1164" 0x00007f6217dadad3 in ?? ()
33 LWP 11648 "rpc worker-1164" 0x00007f6217dadad3 in ?? ()
34 LWP 11649 "rpc worker-1164" 0x00007f6217dadad3 in ?? ()
35 LWP 11650 "rpc worker-1165" 0x00007f6217dadad3 in ?? ()
36 LWP 11651 "rpc worker-1165" 0x00007f6217dadad3 in ?? ()
37 LWP 11652 "rpc worker-1165" 0x00007f6217dadad3 in ?? ()
38 LWP 11653 "rpc worker-1165" 0x00007f6217dadad3 in ?? ()
39 LWP 11654 "rpc worker-1165" 0x00007f6217dadad3 in ?? ()
40 LWP 11655 "rpc worker-1165" 0x00007f6217dadad3 in ?? ()
41 LWP 11656 "rpc worker-1165" 0x00007f6217dadad3 in ?? ()
42 LWP 11657 "rpc worker-1165" 0x00007f6217dadad3 in ?? ()
43 LWP 11658 "rpc worker-1165" 0x00007f6217dadad3 in ?? ()
44 LWP 11659 "rpc worker-1165" 0x00007f6217dadad3 in ?? ()
45 LWP 11660 "rpc worker-1166" 0x00007f6217dadad3 in ?? ()
46 LWP 11661 "rpc worker-1166" 0x00007f6217dadad3 in ?? ()
47 LWP 11662 "rpc worker-1166" 0x00007f6217dadad3 in ?? ()
48 LWP 11663 "rpc worker-1166" 0x00007f6217dadad3 in ?? ()
49 LWP 11664 "rpc worker-1166" 0x00007f6217dadad3 in ?? ()
50 LWP 11665 "rpc worker-1166" 0x00007f6217dadad3 in ?? ()
51 LWP 11666 "rpc worker-1166" 0x00007f6217dadad3 in ?? ()
52 LWP 11667 "rpc worker-1166" 0x00007f6217dadad3 in ?? ()
53 LWP 11668 "rpc worker-1166" 0x00007f6217dadad3 in ?? ()
54 LWP 11669 "rpc worker-1166" 0x00007f6217dadad3 in ?? ()
55 LWP 11670 "rpc worker-1167" 0x00007f6217dadad3 in ?? ()
56 LWP 11671 "rpc worker-1167" 0x00007f6217dadad3 in ?? ()
57 LWP 11672 "rpc worker-1167" 0x00007f6217dadad3 in ?? ()
58 LWP 11673 "rpc worker-1167" 0x00007f6217dadad3 in ?? ()
59 LWP 11674 "rpc worker-1167" 0x00007f6217dadad3 in ?? ()
60 LWP 11675 "rpc worker-1167" 0x00007f6217dadad3 in ?? ()
61 LWP 11676 "rpc worker-1167" 0x00007f6217dadad3 in ?? ()
62 LWP 11677 "rpc worker-1167" 0x00007f6217dadad3 in ?? ()
63 LWP 11678 "rpc worker-1167" 0x00007f6217dadad3 in ?? ()
64 LWP 11679 "rpc worker-1167" 0x00007f6217dadad3 in ?? ()
65 LWP 11680 "rpc worker-1168" 0x00007f6217dadad3 in ?? ()
66 LWP 11681 "rpc worker-1168" 0x00007f6217dadad3 in ?? ()
67 LWP 11682 "rpc worker-1168" 0x00007f6217dadad3 in ?? ()
68 LWP 11683 "rpc worker-1168" 0x00007f6217dadad3 in ?? ()
69 LWP 11684 "rpc worker-1168" 0x00007f6217dadad3 in ?? ()
70 LWP 11685 "rpc worker-1168" 0x00007f6217dadad3 in ?? ()
71 LWP 11686 "rpc worker-1168" 0x00007f6217dadad3 in ?? ()
72 LWP 11687 "rpc worker-1168" 0x00007f6217dadad3 in ?? ()
73 LWP 11688 "rpc worker-1168" 0x00007f6217dadad3 in ?? ()
74 LWP 11689 "rpc worker-1168" 0x00007f6217dadad3 in ?? ()
75 LWP 11690 "rpc worker-1169" 0x00007f6217dadad3 in ?? ()
76 LWP 11691 "rpc worker-1169" 0x00007f6217dadad3 in ?? ()
77 LWP 11692 "rpc worker-1169" 0x00007f6217dadad3 in ?? ()
78 LWP 11693 "rpc worker-1169" 0x00007f6217dadad3 in ?? ()
79 LWP 11694 "rpc worker-1169" 0x00007f6217dadad3 in ?? ()
80 LWP 11695 "rpc worker-1169" 0x00007f6217dadad3 in ?? ()
81 LWP 11696 "rpc worker-1169" 0x00007f6217dadad3 in ?? ()
82 LWP 11697 "rpc worker-1169" 0x00007f6217dadad3 in ?? ()
83 LWP 11698 "rpc worker-1169" 0x00007f6217dadad3 in ?? ()
84 LWP 11699 "rpc worker-1169" 0x00007f6217dadad3 in ?? ()
85 LWP 11700 "rpc worker-1170" 0x00007f6217dadad3 in ?? ()
86 LWP 11701 "rpc worker-1170" 0x00007f6217dadad3 in ?? ()
87 LWP 11702 "rpc worker-1170" 0x00007f6217dadad3 in ?? ()
88 LWP 11703 "rpc worker-1170" 0x00007f6217dadad3 in ?? ()
89 LWP 11704 "rpc worker-1170" 0x00007f6217dadad3 in ?? ()
90 LWP 11705 "rpc worker-1170" 0x00007f6217dadad3 in ?? ()
91 LWP 11706 "rpc worker-1170" 0x00007f6217dadad3 in ?? ()
92 LWP 11707 "rpc worker-1170" 0x00007f6217dadad3 in ?? ()
93 LWP 11708 "rpc worker-1170" 0x00007f6217dadad3 in ?? ()
94 LWP 11709 "rpc worker-1170" 0x00007f6217dadad3 in ?? ()
95 LWP 11710 "rpc worker-1171" 0x00007f6217dadad3 in ?? ()
96 LWP 11711 "rpc worker-1171" 0x00007f6217dadad3 in ?? ()
97 LWP 11712 "rpc worker-1171" 0x00007f6217dadad3 in ?? ()
98 LWP 11713 "rpc worker-1171" 0x00007f6217dadad3 in ?? ()
99 LWP 11714 "rpc worker-1171" 0x00007f6217dadad3 in ?? ()
100 LWP 11715 "rpc worker-1171" 0x00007f6217dadad3 in ?? ()
101 LWP 11716 "rpc worker-1171" 0x00007f6217dadad3 in ?? ()
102 LWP 11717 "rpc worker-1171" 0x00007f6217dadad3 in ?? ()
103 LWP 11718 "rpc worker-1171" 0x00007f6217dadad3 in ?? ()
104 LWP 11719 "rpc worker-1171" 0x00007f6217dadad3 in ?? ()
105 LWP 11720 "rpc worker-1172" 0x00007f6217dadad3 in ?? ()
106 LWP 11721 "rpc worker-1172" 0x00007f6217dadad3 in ?? ()
107 LWP 11722 "rpc worker-1172" 0x00007f6217dadad3 in ?? ()
108 LWP 11723 "rpc worker-1172" 0x00007f6217dadad3 in ?? ()
109 LWP 11724 "rpc worker-1172" 0x00007f6217dadad3 in ?? ()
110 LWP 11725 "rpc worker-1172" 0x00007f6217dadad3 in ?? ()
111 LWP 11726 "rpc worker-1172" 0x00007f6217dadad3 in ?? ()
112 LWP 11727 "rpc worker-1172" 0x00007f6217dadad3 in ?? ()
113 LWP 11728 "rpc worker-1172" 0x00007f6217dadad3 in ?? ()
114 LWP 11729 "rpc worker-1172" 0x00007f6217dadad3 in ?? ()
115 LWP 11730 "rpc worker-1173" 0x00007f6217dadad3 in ?? ()
116 LWP 11731 "rpc worker-1173" 0x00007f6217dadad3 in ?? ()
117 LWP 11732 "diag-logger-117" 0x00007f6217dadfb9 in ?? ()
118 LWP 11733 "result-tracker-" 0x00007f6217dadfb9 in ?? ()
119 LWP 11734 "excess-log-dele" 0x00007f6217dadfb9 in ?? ()
120 LWP 11735 "acceptor-11735" 0x00007f6210b190c7 in ?? ()
121 LWP 11736 "heartbeat-11736" 0x00007f6217dadfb9 in ?? ()
122 LWP 11737 "maintenance_sch" 0x00007f6217dadfb9 in ?? ()
Thread 122 (LWP 11737):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000083 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f61cc621550 in ?? ()
#6 0x0000000000000106 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 11736):
#0 0x00007f6217dadfb9 in ?? ()
#1 0xcf5ed54e759050c7 in ?? ()
#2 0x0000000000000023 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061300001aac4 in ?? ()
#5 0x00007f61cce495b0 in ?? ()
#6 0x0000000000000047 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f61cce495d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007f61cce49650 in ?? ()
#12 0x00000fec399c92cc in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 11735):
#0 0x00007f6210b190c7 in ?? ()
#1 0x00006140000218a8 in ?? ()
#2 0x00007f61cd660d50 in ?? ()
#3 0x00007f61cd660d80 in ?? ()
#4 0x00007f61cd660e80 in ?? ()
#5 0x00007f61cd660d70 in ?? ()
#6 0x00007f61cd660de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0xffffffffffffffff in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 119 (LWP 11734):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x00007f61cde79f00 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 11733):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x00007f61ce6930c0 in ?? ()
#2 0x0000000000000020 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f61ce6930b0 in ?? ()
#6 0x0000000000000040 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 11732):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 11731):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b6848 in ?? ()
#4 0x00007f61cf6c2e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61cf6c2e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 11730):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 11729):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 11728):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 11727):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 11726):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 11725):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 11724):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 11723):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 11722):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 11721):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 11720):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 11719):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 11718):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 11717):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 11716):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 11715):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 11714):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 11713):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 11712):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 11711):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 11710):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 11709):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 11708):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 11707):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 11706):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 11705):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 11704):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 11703):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 11702):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 11701):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 11700):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 11699):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 11698):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 11697):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 11696):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 11695):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 11694):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 11693):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 11692):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 11691):
#0 0x00007f6217dadad3 in ?? ()
#1 0x00000000000002d9 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001b007c in ?? ()
#4 0x00007f61e3a93e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61e3a93e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001b0030 in ?? ()
#9 0x00007f6217dad770 in ?? ()
#10 0x00007f61e3a93e70 in ?? ()
#11 0x00000fec3c7527d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 11690):
#0 0x00007f6217dadad3 in ?? ()
#1 0x00000000000002fa in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007f61e42abe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61e42abe70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 11689):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000007 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007f61e4ac3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61e4ac3e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007f6217dad770 in ?? ()
#10 0x00007f61e4ac3e70 in ?? ()
#11 0x00000fec3c9587d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 73 (LWP 11688):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 11687):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 11686):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 11685):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 11684):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 11683):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 11682):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 11681):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 11680):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 11679):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 11678):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 11677):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 11676):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 11675):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 11674):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 11673):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 11672):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 11671):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000130028 in ?? ()
#4 0x00007f61edc74e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61edc74e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 11670):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 11669):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 11668):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 11667):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 11666):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 11665):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 11664):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 11663):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 11662):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 11661):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 11660):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 11659):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 11658):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 11657):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 11656):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 11655):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 11654):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 11653):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 11652):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 11651):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000acfe8 in ?? ()
#4 0x00007f61f7e55e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61f7e55e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 11650):
#0 0x00007f6217dadad3 in ?? ()
#1 0x000000000000008b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f61f866de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61f866de70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f6217dad770 in ?? ()
#10 0x00007f61f866de70 in ?? ()
#11 0x00000fec3f0cdbd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 11649):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 11648):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 11647):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 11646):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 11645):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 11644):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 11643):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 11642):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 11641):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 11640):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 11639):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 11638):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 11637):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 11636):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 11635):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 11634):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 11633):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 11632):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 11631):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x00007f6202066c80 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007f6202066c70 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 11630):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 11628):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x00007f62038af1c0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007f62038af1b0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 11627):
#0 0x00007f6217dadad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 11626):
#0 0x00007f6210b17a47 in ?? ()
#1 0x00007f62048b22e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f62048b22d0 in ?? ()
#4 0x00007f62048b24e0 in ?? ()
#5 0x00007f62048b2320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007f62048b23a0 in ?? ()
#8 0x00007f621317725d in ?? ()
#9 0x3fb96dbbebed8000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x00000000db000000 in ?? ()
#14 0x00007f6200000000 in ?? ()
#15 0x41d9e134dfe3c36c in ?? ()
#16 0x00000fecc090e474 in ?? ()
#17 0x00007f62048b2380 in ?? ()
#18 0x00007f621317bba3 in ?? ()
#19 0x00007f62048b2350 in ?? ()
#20 0x3fb96dbbebed8000 in ?? ()
#21 0x00000000048b23a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb96dbbebed8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 11625):
#0 0x00007f6210b17a47 in ?? ()
#1 0x00007f62050b32e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f62050b32d0 in ?? ()
#4 0x00007f62050b34e0 in ?? ()
#5 0x00007f62050b3320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007f62050b33a0 in ?? ()
#8 0x00007f621317725d in ?? ()
#9 0x3fb969d19566c000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x00000000fe000000 in ?? ()
#14 0x00007f6200000000 in ?? ()
#15 0x41d9e134dfe3c36b in ?? ()
#16 0x00000fecc0a0e674 in ?? ()
#17 0x00007f62050b3380 in ?? ()
#18 0x00007f621317bba3 in ?? ()
#19 0x00007f62050b3350 in ?? ()
#20 0x3fb969d19566c000 in ?? ()
#21 0x00000000050b33a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb969d19566c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 11624):
#0 0x00007f6210b17a47 in ?? ()
#1 0x00007f62058b42e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f62058b42d0 in ?? ()
#4 0x00007f62058b44e0 in ?? ()
#5 0x00007f62058b4320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f62058b43a0 in ?? ()
#8 0x00007f621317725d in ?? ()
#9 0x3fb97d04a750c000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000084800000 in ?? ()
#14 0x00007f6200000000 in ?? ()
#15 0x41d9e134dfe3c36a in ?? ()
#16 0x00000fecc0b0e874 in ?? ()
#17 0x00007f62058b4380 in ?? ()
#18 0x00007f621317bba3 in ?? ()
#19 0x00007f62058b4350 in ?? ()
#20 0x3fb97d04a750c000 in ?? ()
#21 0x00000000058b43a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb97d04a750c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 11623):
#0 0x00007f6210b17a47 in ?? ()
#1 0x00007f62078ad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f62078ad2d0 in ?? ()
#4 0x00007f62078ad4e0 in ?? ()
#5 0x00007f62078ad320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f62078ad3a0 in ?? ()
#8 0x00007f621317725d in ?? ()
#9 0x3fb9836d0bbe2000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x000000005bc00000 in ?? ()
#14 0x00007f6200000000 in ?? ()
#15 0x41d9e134dfe3c36d in ?? ()
#16 0x00000fecc0f0da74 in ?? ()
#17 0x00007f62078ad380 in ?? ()
#18 0x00007f621317bba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 11620):
#0 0x00007f6210b0acb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f62092b7788 in ?? ()
#3 0x0000602000015a10 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 11619):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x00007f6208ab6080 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 11618):
#0 0x00007f6217db19e2 in ?? ()
#1 0x00007f62082b3b60 in ?? ()
#2 0x00007f62082b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f62082b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f62082b3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f62082b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f621d6d5f0c in ?? ()
#12 0x00007f621d6c57b9 in ?? ()
#13 0x00007f621d6c9da6 in ?? ()
#14 0x00007f621d6ce7e0 in ?? ()
#15 0x00007f621d6ce609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f62149c40be in ?? ()
#18 0x00007f6217da76db in ?? ()
#19 0x00000fecc104e67c in ?? ()
#20 0x00007f62082b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 11612):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x00007f620a2b8c40 in ?? ()
#2 0x00000000000000a5 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f620a2b8c30 in ?? ()
#6 0x000000000000014a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 11611):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 11610):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 11609):
#0 0x00007f6217dadfb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f620babc410 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 11607):
#0 0x00007f6217db1d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 08:56:53.865687 7660 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID cf369533d8ad453f9d6c47cad3706dab and pid 12017
************************ BEGIN STACKS **************************
[New LWP 12019]
[New LWP 12020]
[New LWP 12021]
[New LWP 12022]
[New LWP 12028]
[New LWP 12029]
[New LWP 12030]
[New LWP 12033]
[New LWP 12034]
[New LWP 12035]
[New LWP 12036]
[New LWP 12037]
[New LWP 12038]
[New LWP 12040]
[New LWP 12041]
[New LWP 12042]
[New LWP 12043]
[New LWP 12044]
[New LWP 12045]
[New LWP 12046]
[New LWP 12047]
[New LWP 12048]
[New LWP 12049]
[New LWP 12050]
[New LWP 12051]
[New LWP 12052]
[New LWP 12053]
[New LWP 12054]
[New LWP 12055]
[New LWP 12056]
[New LWP 12057]
[New LWP 12058]
[New LWP 12059]
[New LWP 12060]
[New LWP 12061]
[New LWP 12062]
[New LWP 12063]
[New LWP 12064]
[New LWP 12065]
[New LWP 12066]
[New LWP 12067]
[New LWP 12068]
[New LWP 12069]
[New LWP 12070]
[New LWP 12071]
[New LWP 12072]
[New LWP 12073]
[New LWP 12074]
[New LWP 12075]
[New LWP 12076]
[New LWP 12077]
[New LWP 12078]
[New LWP 12079]
[New LWP 12080]
[New LWP 12081]
[New LWP 12082]
[New LWP 12083]
[New LWP 12084]
[New LWP 12085]
[New LWP 12086]
[New LWP 12087]
[New LWP 12088]
[New LWP 12089]
[New LWP 12090]
[New LWP 12091]
[New LWP 12092]
[New LWP 12093]
[New LWP 12094]
[New LWP 12095]
[New LWP 12096]
[New LWP 12097]
[New LWP 12098]
[New LWP 12099]
[New LWP 12100]
[New LWP 12101]
[New LWP 12102]
[New LWP 12103]
[New LWP 12104]
[New LWP 12105]
[New LWP 12106]
[New LWP 12107]
[New LWP 12108]
[New LWP 12109]
[New LWP 12110]
[New LWP 12111]
[New LWP 12112]
[New LWP 12113]
[New LWP 12114]
[New LWP 12115]
[New LWP 12116]
[New LWP 12117]
[New LWP 12118]
[New LWP 12119]
[New LWP 12120]
[New LWP 12121]
[New LWP 12122]
[New LWP 12123]
[New LWP 12124]
[New LWP 12125]
[New LWP 12126]
[New LWP 12127]
[New LWP 12128]
[New LWP 12129]
[New LWP 12130]
[New LWP 12131]
[New LWP 12132]
[New LWP 12133]
[New LWP 12134]
[New LWP 12135]
[New LWP 12136]
[New LWP 12137]
[New LWP 12138]
[New LWP 12139]
[New LWP 12140]
[New LWP 12141]
[New LWP 12142]
[New LWP 12143]
[New LWP 12144]
[New LWP 12145]
[New LWP 12146]
[New LWP 12147]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f4900393d50 in ?? ()
Id Target Id Frame
* 1 LWP 12017 "kudu" 0x00007f4900393d50 in ?? ()
2 LWP 12019 "kudu" 0x00007f490038ffb9 in ?? ()
3 LWP 12020 "kudu" 0x00007f490038ffb9 in ?? ()
4 LWP 12021 "kudu" 0x00007f490038ffb9 in ?? ()
5 LWP 12022 "kernel-watcher-" 0x00007f490038ffb9 in ?? ()
6 LWP 12028 "ntp client-1202" 0x00007f49003939e2 in ?? ()
7 LWP 12029 "file cache-evic" 0x00007f490038ffb9 in ?? ()
8 LWP 12030 "sq_acceptor" 0x00007f48f90eccb9 in ?? ()
9 LWP 12033 "rpc reactor-120" 0x00007f48f90f9a47 in ?? ()
10 LWP 12034 "rpc reactor-120" 0x00007f48f90f9a47 in ?? ()
11 LWP 12035 "rpc reactor-120" 0x00007f48f90f9a47 in ?? ()
12 LWP 12036 "rpc reactor-120" 0x00007f48f90f9a47 in ?? ()
13 LWP 12037 "MaintenanceMgr " 0x00007f490038fad3 in ?? ()
14 LWP 12038 "txn-status-mana" 0x00007f490038ffb9 in ?? ()
15 LWP 12040 "collect_and_rem" 0x00007f490038ffb9 in ?? ()
16 LWP 12041 "tc-session-exp-" 0x00007f490038ffb9 in ?? ()
17 LWP 12042 "rpc worker-1204" 0x00007f490038fad3 in ?? ()
18 LWP 12043 "rpc worker-1204" 0x00007f490038fad3 in ?? ()
19 LWP 12044 "rpc worker-1204" 0x00007f490038fad3 in ?? ()
20 LWP 12045 "rpc worker-1204" 0x00007f490038fad3 in ?? ()
21 LWP 12046 "rpc worker-1204" 0x00007f490038fad3 in ?? ()
22 LWP 12047 "rpc worker-1204" 0x00007f490038fad3 in ?? ()
23 LWP 12048 "rpc worker-1204" 0x00007f490038fad3 in ?? ()
24 LWP 12049 "rpc worker-1204" 0x00007f490038fad3 in ?? ()
25 LWP 12050 "rpc worker-1205" 0x00007f490038fad3 in ?? ()
26 LWP 12051 "rpc worker-1205" 0x00007f490038fad3 in ?? ()
27 LWP 12052 "rpc worker-1205" 0x00007f490038fad3 in ?? ()
28 LWP 12053 "rpc worker-1205" 0x00007f490038fad3 in ?? ()
29 LWP 12054 "rpc worker-1205" 0x00007f490038fad3 in ?? ()
30 LWP 12055 "rpc worker-1205" 0x00007f490038fad3 in ?? ()
31 LWP 12056 "rpc worker-1205" 0x00007f490038fad3 in ?? ()
32 LWP 12057 "rpc worker-1205" 0x00007f490038fad3 in ?? ()
33 LWP 12058 "rpc worker-1205" 0x00007f490038fad3 in ?? ()
34 LWP 12059 "rpc worker-1205" 0x00007f490038fad3 in ?? ()
35 LWP 12060 "rpc worker-1206" 0x00007f490038fad3 in ?? ()
36 LWP 12061 "rpc worker-1206" 0x00007f490038fad3 in ?? ()
37 LWP 12062 "rpc worker-1206" 0x00007f490038fad3 in ?? ()
38 LWP 12063 "rpc worker-1206" 0x00007f490038fad3 in ?? ()
39 LWP 12064 "rpc worker-1206" 0x00007f490038fad3 in ?? ()
40 LWP 12065 "rpc worker-1206" 0x00007f490038fad3 in ?? ()
41 LWP 12066 "rpc worker-1206" 0x00007f490038fad3 in ?? ()
42 LWP 12067 "rpc worker-1206" 0x00007f490038fad3 in ?? ()
43 LWP 12068 "rpc worker-1206" 0x00007f490038fad3 in ?? ()
44 LWP 12069 "rpc worker-1206" 0x00007f490038fad3 in ?? ()
45 LWP 12070 "rpc worker-1207" 0x00007f490038fad3 in ?? ()
46 LWP 12071 "rpc worker-1207" 0x00007f490038fad3 in ?? ()
47 LWP 12072 "rpc worker-1207" 0x00007f490038fad3 in ?? ()
48 LWP 12073 "rpc worker-1207" 0x00007f490038fad3 in ?? ()
49 LWP 12074 "rpc worker-1207" 0x00007f490038fad3 in ?? ()
50 LWP 12075 "rpc worker-1207" 0x00007f490038fad3 in ?? ()
51 LWP 12076 "rpc worker-1207" 0x00007f490038fad3 in ?? ()
52 LWP 12077 "rpc worker-1207" 0x00007f490038fad3 in ?? ()
53 LWP 12078 "rpc worker-1207" 0x00007f490038fad3 in ?? ()
54 LWP 12079 "rpc worker-1207" 0x00007f490038fad3 in ?? ()
55 LWP 12080 "rpc worker-1208" 0x00007f490038fad3 in ?? ()
56 LWP 12081 "rpc worker-1208" 0x00007f490038fad3 in ?? ()
57 LWP 12082 "rpc worker-1208" 0x00007f490038fad3 in ?? ()
58 LWP 12083 "rpc worker-1208" 0x00007f490038fad3 in ?? ()
59 LWP 12084 "rpc worker-1208" 0x00007f490038fad3 in ?? ()
60 LWP 12085 "rpc worker-1208" 0x00007f490038fad3 in ?? ()
61 LWP 12086 "rpc worker-1208" 0x00007f490038fad3 in ?? ()
62 LWP 12087 "rpc worker-1208" 0x00007f490038fad3 in ?? ()
63 LWP 12088 "rpc worker-1208" 0x00007f490038fad3 in ?? ()
64 LWP 12089 "rpc worker-1208" 0x00007f490038fad3 in ?? ()
65 LWP 12090 "rpc worker-1209" 0x00007f490038fad3 in ?? ()
66 LWP 12091 "rpc worker-1209" 0x00007f490038fad3 in ?? ()
67 LWP 12092 "rpc worker-1209" 0x00007f490038fad3 in ?? ()
68 LWP 12093 "rpc worker-1209" 0x00007f490038fad3 in ?? ()
69 LWP 12094 "rpc worker-1209" 0x00007f490038fad3 in ?? ()
70 LWP 12095 "rpc worker-1209" 0x00007f490038fad3 in ?? ()
71 LWP 12096 "rpc worker-1209" 0x00007f490038fad3 in ?? ()
72 LWP 12097 "rpc worker-1209" 0x00007f490038fad3 in ?? ()
73 LWP 12098 "rpc worker-1209" 0x00007f490038fad3 in ?? ()
74 LWP 12099 "rpc worker-1209" 0x00007f490038fad3 in ?? ()
75 LWP 12100 "rpc worker-1210" 0x00007f490038fad3 in ?? ()
76 LWP 12101 "rpc worker-1210" 0x00007f490038fad3 in ?? ()
77 LWP 12102 "rpc worker-1210" 0x00007f490038fad3 in ?? ()
78 LWP 12103 "rpc worker-1210" 0x00007f490038fad3 in ?? ()
79 LWP 12104 "rpc worker-1210" 0x00007f490038fad3 in ?? ()
80 LWP 12105 "rpc worker-1210" 0x00007f490038fad3 in ?? ()
81 LWP 12106 "rpc worker-1210" 0x00007f490038fad3 in ?? ()
82 LWP 12107 "rpc worker-1210" 0x00007f490038fad3 in ?? ()
83 LWP 12108 "rpc worker-1210" 0x00007f490038fad3 in ?? ()
84 LWP 12109 "rpc worker-1210" 0x00007f490038fad3 in ?? ()
85 LWP 12110 "rpc worker-1211" 0x00007f490038fad3 in ?? ()
86 LWP 12111 "rpc worker-1211" 0x00007f490038fad3 in ?? ()
87 LWP 12112 "rpc worker-1211" 0x00007f490038fad3 in ?? ()
88 LWP 12113 "rpc worker-1211" 0x00007f490038fad3 in ?? ()
89 LWP 12114 "rpc worker-1211" 0x00007f490038fad3 in ?? ()
90 LWP 12115 "rpc worker-1211" 0x00007f490038fad3 in ?? ()
91 LWP 12116 "rpc worker-1211" 0x00007f490038fad3 in ?? ()
92 LWP 12117 "rpc worker-1211" 0x00007f490038fad3 in ?? ()
93 LWP 12118 "rpc worker-1211" 0x00007f490038fad3 in ?? ()
94 LWP 12119 "rpc worker-1211" 0x00007f490038fad3 in ?? ()
95 LWP 12120 "rpc worker-1212" 0x00007f490038fad3 in ?? ()
96 LWP 12121 "rpc worker-1212" 0x00007f490038fad3 in ?? ()
97 LWP 12122 "rpc worker-1212" 0x00007f490038fad3 in ?? ()
98 LWP 12123 "rpc worker-1212" 0x00007f490038fad3 in ?? ()
99 LWP 12124 "rpc worker-1212" 0x00007f490038fad3 in ?? ()
100 LWP 12125 "rpc worker-1212" 0x00007f490038fad3 in ?? ()
101 LWP 12126 "rpc worker-1212" 0x00007f490038fad3 in ?? ()
102 LWP 12127 "rpc worker-1212" 0x00007f490038fad3 in ?? ()
103 LWP 12128 "rpc worker-1212" 0x00007f490038fad3 in ?? ()
104 LWP 12129 "rpc worker-1212" 0x00007f490038fad3 in ?? ()
105 LWP 12130 "rpc worker-1213" 0x00007f490038fad3 in ?? ()
106 LWP 12131 "rpc worker-1213" 0x00007f490038fad3 in ?? ()
107 LWP 12132 "rpc worker-1213" 0x00007f490038fad3 in ?? ()
108 LWP 12133 "rpc worker-1213" 0x00007f490038fad3 in ?? ()
109 LWP 12134 "rpc worker-1213" 0x00007f490038fad3 in ?? ()
110 LWP 12135 "rpc worker-1213" 0x00007f490038fad3 in ?? ()
111 LWP 12136 "rpc worker-1213" 0x00007f490038fad3 in ?? ()
112 LWP 12137 "rpc worker-1213" 0x00007f490038fad3 in ?? ()
113 LWP 12138 "rpc worker-1213" 0x00007f490038fad3 in ?? ()
114 LWP 12139 "rpc worker-1213" 0x00007f490038fad3 in ?? ()
115 LWP 12140 "rpc worker-1214" 0x00007f490038fad3 in ?? ()
116 LWP 12141 "rpc worker-1214" 0x00007f490038fad3 in ?? ()
117 LWP 12142 "diag-logger-121" 0x00007f490038ffb9 in ?? ()
118 LWP 12143 "result-tracker-" 0x00007f490038ffb9 in ?? ()
119 LWP 12144 "excess-log-dele" 0x00007f490038ffb9 in ?? ()
120 LWP 12145 "acceptor-12145" 0x00007f48f90fb0c7 in ?? ()
121 LWP 12146 "heartbeat-12146" 0x00007f490038ffb9 in ?? ()
122 LWP 12147 "maintenance_sch" 0x00007f490038ffb9 in ?? ()
Thread 122 (LWP 12147):
#0 0x00007f490038ffb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000072 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f48b541b550 in ?? ()
#6 0x00000000000000e4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 12146):
#0 0x00007f490038ffb9 in ?? ()
#1 0xcf5ed54e759050c7 in ?? ()
#2 0x000000000000001d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007f48b5c335b0 in ?? ()
#6 0x000000000000003a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 12145):
#0 0x00007f48f90fb0c7 in ?? ()
#1 0x0000614000021ca8 in ?? ()
#2 0x00007f48b645ad50 in ?? ()
#3 0x00007f48b645ad80 in ?? ()
#4 0x00007f48b645ae80 in ?? ()
#5 0x00007f48b645ad70 in ?? ()
#6 0x00007f48b645ade0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007f48b645af00 in ?? ()
#11 0x00007f48b645a63c in ?? ()
#12 0x00000032b645a5b0 in ?? ()
#13 0x00007f48b5c5e000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 12144):
#0 0x00007f490038ffb9 in ?? ()
#1 0x00007f48b6c73f00 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 12143):
#0 0x00007f490038ffb9 in ?? ()
#1 0x00007f48b748d0c0 in ?? ()
#2 0x000000000000001c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f48b748d0b0 in ?? ()
#6 0x0000000000000038 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 12142):
#0 0x00007f490038ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 12141):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002d682c in ?? ()
#4 0x00007f48b84bce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48b84bce70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002d67e0 in ?? ()
#9 0x00007f490038f770 in ?? ()
#10 0x00007f48b84bce70 in ?? ()
#11 0x00000fe9170979d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 12140):
#0 0x00007f490038fad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002d002c in ?? ()
#4 0x00007f48b8cd4e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48b8cd4e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002cffe0 in ?? ()
#9 0x00007f490038f770 in ?? ()
#10 0x00007f48b8cd4e70 in ?? ()
#11 0x00000fe91719a9d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 12139):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 12138):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 12137):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 12136):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 12135):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 12134):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 12133):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 12132):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 12131):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 12130):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 12129):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 12128):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 12127):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 12126):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 12125):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 12124):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 12123):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 12122):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 12121):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 12120):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 12119):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 12118):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 12117):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 12116):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 12115):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 12114):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 12113):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 12112):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 12111):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 12110):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 12109):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 12108):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 12107):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 12106):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 12105):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 12104):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 12103):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 12102):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 12101):
#0 0x00007f490038fad3 in ?? ()
#1 0x00000000000001b0 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001c6868 in ?? ()
#4 0x00007f48cc075e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48cc075e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 12100):
#0 0x00007f490038fad3 in ?? ()
#1 0x000000000000021e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001c0068 in ?? ()
#4 0x00007f48cc88de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48cc88de70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 12099):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000064 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001b0078 in ?? ()
#4 0x00007f48cd0a5e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48cd0a5e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 12098):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 12097):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 12096):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 12095):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 12094):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 12093):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 12092):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 12091):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 12090):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 12089):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 12088):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 12087):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 12086):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 12085):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 12084):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 12083):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 12082):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 12081):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00013d028 in ?? ()
#4 0x00007f48d6256e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48d6256e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 12080):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 12079):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 12078):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 12077):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 12076):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 12075):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 12074):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 12073):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 12072):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 12071):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 12070):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 12069):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 12068):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 12067):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 12066):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 12065):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 12064):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 12063):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 12062):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 12061):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000059 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000affdc in ?? ()
#4 0x00007f48e0437e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48e0437e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000aff90 in ?? ()
#9 0x00007f490038f770 in ?? ()
#10 0x00007f48e0437e70 in ?? ()
#11 0x00000fe91c086fd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 12060):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000acfec in ?? ()
#4 0x00007f48e0c4fe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48e0c4fe70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000acfa0 in ?? ()
#9 0x00007f490038f770 in ?? ()
#10 0x00007f48e0c4fe70 in ?? ()
#11 0x00000fe91c189fd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 12059):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 12058):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 12057):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 12056):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 12055):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 12054):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 12053):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 12052):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 12051):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 12050):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 12049):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 12048):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 12047):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 12046):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 12045):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 12044):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 12043):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 12042):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 12041):
#0 0x00007f490038ffb9 in ?? ()
#1 0x00007f48ea638c80 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007f48ea638c70 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 12040):
#0 0x00007f490038ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 12038):
#0 0x00007f490038ffb9 in ?? ()
#1 0x00007f48ebeaf1c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007f48ebeaf1b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 12037):
#0 0x00007f490038fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 12036):
#0 0x00007f48f90f9a47 in ?? ()
#1 0x00007f48eceb22e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f48eceb22d0 in ?? ()
#4 0x00007f48eceb24e0 in ?? ()
#5 0x00007f48eceb2320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007f48eceb23a0 in ?? ()
#8 0x00007f48fb75925d in ?? ()
#9 0x3fb9659cec2aa000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x00000000fc800000 in ?? ()
#14 0x00007f4800000000 in ?? ()
#15 0x41d9e134dfe3c367 in ?? ()
#16 0x00000fe99d9ce474 in ?? ()
#17 0x00007f48eceb2380 in ?? ()
#18 0x00007f48fb75dba3 in ?? ()
#19 0x00007f48eceb2350 in ?? ()
#20 0x3fb9659cec2aa000 in ?? ()
#21 0x00000000eceb23a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb9659cec2aa000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 12035):
#0 0x00007f48f90f9a47 in ?? ()
#1 0x00007f48ed6b32e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f48ed6b32d0 in ?? ()
#4 0x00007f48ed6b34e0 in ?? ()
#5 0x00007f48ed6b3320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007f48ed6b33a0 in ?? ()
#8 0x00007f48fb75925d in ?? ()
#9 0x3fb97b6a9e4fa000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000014000000 in ?? ()
#14 0x00007f4800000000 in ?? ()
#15 0x41d9e134dfe3c368 in ?? ()
#16 0x00000fe99dace674 in ?? ()
#17 0x00007f48ed6b3380 in ?? ()
#18 0x00007f48fb75dba3 in ?? ()
#19 0x00007f48ed6b3350 in ?? ()
#20 0x3fb97b6a9e4fa000 in ?? ()
#21 0x00000000ed6b33a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb97b6a9e4fa000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 12034):
#0 0x00007f48f90f9a47 in ?? ()
#1 0x00007f48edeb42e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f48edeb42d0 in ?? ()
#4 0x00007f48edeb44e0 in ?? ()
#5 0x00007f48edeb4320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f48edeb43a0 in ?? ()
#8 0x00007f48fb75925d in ?? ()
#9 0x3fa53cb842350000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x00000000f0500000 in ?? ()
#14 0x00007f4800000000 in ?? ()
#15 0x41d9e134dfe3c368 in ?? ()
#16 0x00000fe99dbce874 in ?? ()
#17 0x00007f48edeb4380 in ?? ()
#18 0x00007f48fb75dba3 in ?? ()
#19 0x00007f48edeb4350 in ?? ()
#20 0x3fa53cb842350000 in ?? ()
#21 0x00000000edeb43a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fa53cb842350000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 12033):
#0 0x00007f48f90f9a47 in ?? ()
#1 0x00007f48efead2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f48efead2d0 in ?? ()
#4 0x00007f48efead4e0 in ?? ()
#5 0x00007f48efead320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f48efead3a0 in ?? ()
#8 0x00007f48fb75925d in ?? ()
#9 0x3fb95d391a741000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x0000000073800000 in ?? ()
#14 0x00007f4800000000 in ?? ()
#15 0x41d9e134dfe3c36c in ?? ()
#16 0x00000fe99dfcda74 in ?? ()
#17 0x00007f48efead380 in ?? ()
#18 0x00007f48fb75dba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 12030):
#0 0x00007f48f90eccb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f48f18b7788 in ?? ()
#3 0x00006020000167d0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 12029):
#0 0x00007f490038ffb9 in ?? ()
#1 0x00007f48f10b6080 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 12028):
#0 0x00007f49003939e2 in ?? ()
#1 0x00007f48f08b3b60 in ?? ()
#2 0x00007f48f08b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f48f08b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f48f08b3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f48f08b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f4905cb7f0c in ?? ()
#12 0x00007f4905ca77b9 in ?? ()
#13 0x00007f4905cabda6 in ?? ()
#14 0x00007f4905cb07e0 in ?? ()
#15 0x00007f4905cb0609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f48fcfa60be in ?? ()
#18 0x00007f49003896db in ?? ()
#19 0x00000fe99e10e67c in ?? ()
#20 0x00007f48f08b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 12022):
#0 0x00007f490038ffb9 in ?? ()
#1 0x00007f48f28b8c40 in ?? ()
#2 0x0000000000000093 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f48f28b8c30 in ?? ()
#6 0x0000000000000126 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 12021):
#0 0x00007f490038ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 12020):
#0 0x00007f490038ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 12019):
#0 0x00007f490038ffb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f48f40bc410 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 12017):
#0 0x00007f4900393d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 08:56:54.702632 7660 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID 00524bc1b1a34b61af38c9f2bded4031 and pid 11882
************************ BEGIN STACKS **************************
[New LWP 11883]
[New LWP 11884]
[New LWP 11885]
[New LWP 11886]
[New LWP 11892]
[New LWP 11893]
[New LWP 11894]
[New LWP 11897]
[New LWP 11898]
[New LWP 11899]
[New LWP 11900]
[New LWP 11902]
[New LWP 11903]
[New LWP 11904]
[New LWP 11905]
[New LWP 11906]
[New LWP 11907]
[New LWP 11908]
[New LWP 11909]
[New LWP 11910]
[New LWP 11911]
[New LWP 11912]
[New LWP 11913]
[New LWP 11914]
[New LWP 11915]
[New LWP 11916]
[New LWP 11917]
[New LWP 11918]
[New LWP 11919]
[New LWP 11920]
[New LWP 11921]
[New LWP 11922]
[New LWP 11923]
[New LWP 11924]
[New LWP 11925]
[New LWP 11926]
[New LWP 11927]
[New LWP 11928]
[New LWP 11929]
[New LWP 11930]
[New LWP 11931]
[New LWP 11932]
[New LWP 11933]
[New LWP 11934]
[New LWP 11935]
[New LWP 11936]
[New LWP 11937]
[New LWP 11938]
[New LWP 11939]
[New LWP 11940]
[New LWP 11941]
[New LWP 11942]
[New LWP 11943]
[New LWP 11944]
[New LWP 11945]
[New LWP 11946]
[New LWP 11947]
[New LWP 11948]
[New LWP 11949]
[New LWP 11950]
[New LWP 11951]
[New LWP 11952]
[New LWP 11953]
[New LWP 11954]
[New LWP 11955]
[New LWP 11956]
[New LWP 11957]
[New LWP 11958]
[New LWP 11959]
[New LWP 11960]
[New LWP 11961]
[New LWP 11962]
[New LWP 11963]
[New LWP 11964]
[New LWP 11965]
[New LWP 11966]
[New LWP 11967]
[New LWP 11968]
[New LWP 11969]
[New LWP 11970]
[New LWP 11971]
[New LWP 11972]
[New LWP 11973]
[New LWP 11974]
[New LWP 11975]
[New LWP 11976]
[New LWP 11977]
[New LWP 11978]
[New LWP 11979]
[New LWP 11980]
[New LWP 11981]
[New LWP 11982]
[New LWP 11983]
[New LWP 11984]
[New LWP 11985]
[New LWP 11986]
[New LWP 11987]
[New LWP 11988]
[New LWP 11989]
[New LWP 11990]
[New LWP 11991]
[New LWP 11992]
[New LWP 11993]
[New LWP 11994]
[New LWP 11995]
[New LWP 11996]
[New LWP 11997]
[New LWP 11998]
[New LWP 11999]
[New LWP 12000]
[New LWP 12001]
[New LWP 12002]
[New LWP 12003]
[New LWP 12004]
[New LWP 12005]
[New LWP 12006]
[New LWP 12007]
[New LWP 12008]
[New LWP 12009]
[New LWP 12010]
[New LWP 12011]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007feafc0bcd50 in ?? ()
Id Target Id Frame
* 1 LWP 11882 "kudu" 0x00007feafc0bcd50 in ?? ()
2 LWP 11883 "kudu" 0x00007feafc0b8fb9 in ?? ()
3 LWP 11884 "kudu" 0x00007feafc0b8fb9 in ?? ()
4 LWP 11885 "kudu" 0x00007feafc0b8fb9 in ?? ()
5 LWP 11886 "kernel-watcher-" 0x00007feafc0b8fb9 in ?? ()
6 LWP 11892 "ntp client-1189" 0x00007feafc0bc9e2 in ?? ()
7 LWP 11893 "file cache-evic" 0x00007feafc0b8fb9 in ?? ()
8 LWP 11894 "sq_acceptor" 0x00007feaf4e15cb9 in ?? ()
9 LWP 11897 "rpc reactor-118" 0x00007feaf4e22a47 in ?? ()
10 LWP 11898 "rpc reactor-118" 0x00007feaf4e22a47 in ?? ()
11 LWP 11899 "rpc reactor-118" 0x00007feaf4e22a47 in ?? ()
12 LWP 11900 "rpc reactor-119" 0x00007feaf4e22a47 in ?? ()
13 LWP 11902 "MaintenanceMgr " 0x00007feafc0b8ad3 in ?? ()
14 LWP 11903 "txn-status-mana" 0x00007feafc0b8fb9 in ?? ()
15 LWP 11904 "collect_and_rem" 0x00007feafc0b8fb9 in ?? ()
16 LWP 11905 "tc-session-exp-" 0x00007feafc0b8fb9 in ?? ()
17 LWP 11906 "rpc worker-1190" 0x00007feafc0b8ad3 in ?? ()
18 LWP 11907 "rpc worker-1190" 0x00007feafc0b8ad3 in ?? ()
19 LWP 11908 "rpc worker-1190" 0x00007feafc0b8ad3 in ?? ()
20 LWP 11909 "rpc worker-1190" 0x00007feafc0b8ad3 in ?? ()
21 LWP 11910 "rpc worker-1191" 0x00007feafc0b8ad3 in ?? ()
22 LWP 11911 "rpc worker-1191" 0x00007feafc0b8ad3 in ?? ()
23 LWP 11912 "rpc worker-1191" 0x00007feafc0b8ad3 in ?? ()
24 LWP 11913 "rpc worker-1191" 0x00007feafc0b8ad3 in ?? ()
25 LWP 11914 "rpc worker-1191" 0x00007feafc0b8ad3 in ?? ()
26 LWP 11915 "rpc worker-1191" 0x00007feafc0b8ad3 in ?? ()
27 LWP 11916 "rpc worker-1191" 0x00007feafc0b8ad3 in ?? ()
28 LWP 11917 "rpc worker-1191" 0x00007feafc0b8ad3 in ?? ()
29 LWP 11918 "rpc worker-1191" 0x00007feafc0b8ad3 in ?? ()
30 LWP 11919 "rpc worker-1191" 0x00007feafc0b8ad3 in ?? ()
31 LWP 11920 "rpc worker-1192" 0x00007feafc0b8ad3 in ?? ()
32 LWP 11921 "rpc worker-1192" 0x00007feafc0b8ad3 in ?? ()
33 LWP 11922 "rpc worker-1192" 0x00007feafc0b8ad3 in ?? ()
34 LWP 11923 "rpc worker-1192" 0x00007feafc0b8ad3 in ?? ()
35 LWP 11924 "rpc worker-1192" 0x00007feafc0b8ad3 in ?? ()
36 LWP 11925 "rpc worker-1192" 0x00007feafc0b8ad3 in ?? ()
37 LWP 11926 "rpc worker-1192" 0x00007feafc0b8ad3 in ?? ()
38 LWP 11927 "rpc worker-1192" 0x00007feafc0b8ad3 in ?? ()
39 LWP 11928 "rpc worker-1192" 0x00007feafc0b8ad3 in ?? ()
40 LWP 11929 "rpc worker-1192" 0x00007feafc0b8ad3 in ?? ()
41 LWP 11930 "rpc worker-1193" 0x00007feafc0b8ad3 in ?? ()
42 LWP 11931 "rpc worker-1193" 0x00007feafc0b8ad3 in ?? ()
43 LWP 11932 "rpc worker-1193" 0x00007feafc0b8ad3 in ?? ()
44 LWP 11933 "rpc worker-1193" 0x00007feafc0b8ad3 in ?? ()
45 LWP 11934 "rpc worker-1193" 0x00007feafc0b8ad3 in ?? ()
46 LWP 11935 "rpc worker-1193" 0x00007feafc0b8ad3 in ?? ()
47 LWP 11936 "rpc worker-1193" 0x00007feafc0b8ad3 in ?? ()
48 LWP 11937 "rpc worker-1193" 0x00007feafc0b8ad3 in ?? ()
49 LWP 11938 "rpc worker-1193" 0x00007feafc0b8ad3 in ?? ()
50 LWP 11939 "rpc worker-1193" 0x00007feafc0b8ad3 in ?? ()
51 LWP 11940 "rpc worker-1194" 0x00007feafc0b8ad3 in ?? ()
52 LWP 11941 "rpc worker-1194" 0x00007feafc0b8ad3 in ?? ()
53 LWP 11942 "rpc worker-1194" 0x00007feafc0b8ad3 in ?? ()
54 LWP 11943 "rpc worker-1194" 0x00007feafc0b8ad3 in ?? ()
55 LWP 11944 "rpc worker-1194" 0x00007feafc0b8ad3 in ?? ()
56 LWP 11945 "rpc worker-1194" 0x00007feafc0b8ad3 in ?? ()
57 LWP 11946 "rpc worker-1194" 0x00007feafc0b8ad3 in ?? ()
58 LWP 11947 "rpc worker-1194" 0x00007feafc0b8ad3 in ?? ()
59 LWP 11948 "rpc worker-1194" 0x00007feafc0b8ad3 in ?? ()
60 LWP 11949 "rpc worker-1194" 0x00007feafc0b8ad3 in ?? ()
61 LWP 11950 "rpc worker-1195" 0x00007feafc0b8ad3 in ?? ()
62 LWP 11951 "rpc worker-1195" 0x00007feafc0b8ad3 in ?? ()
63 LWP 11952 "rpc worker-1195" 0x00007feafc0b8ad3 in ?? ()
64 LWP 11953 "rpc worker-1195" 0x00007feafc0b8ad3 in ?? ()
65 LWP 11954 "rpc worker-1195" 0x00007feafc0b8ad3 in ?? ()
66 LWP 11955 "rpc worker-1195" 0x00007feafc0b8ad3 in ?? ()
67 LWP 11956 "rpc worker-1195" 0x00007feafc0b8ad3 in ?? ()
68 LWP 11957 "rpc worker-1195" 0x00007feafc0b8ad3 in ?? ()
69 LWP 11958 "rpc worker-1195" 0x00007feafc0b8ad3 in ?? ()
70 LWP 11959 "rpc worker-1195" 0x00007feafc0b8ad3 in ?? ()
71 LWP 11960 "rpc worker-1196" 0x00007feafc0b8ad3 in ?? ()
72 LWP 11961 "rpc worker-1196" 0x00007feafc0b8ad3 in ?? ()
73 LWP 11962 "rpc worker-1196" 0x00007feafc0b8ad3 in ?? ()
74 LWP 11963 "rpc worker-1196" 0x00007feafc0b8ad3 in ?? ()
75 LWP 11964 "rpc worker-1196" 0x00007feafc0b8ad3 in ?? ()
76 LWP 11965 "rpc worker-1196" 0x00007feafc0b8ad3 in ?? ()
77 LWP 11966 "rpc worker-1196" 0x00007feafc0b8ad3 in ?? ()
78 LWP 11967 "rpc worker-1196" 0x00007feafc0b8ad3 in ?? ()
79 LWP 11968 "rpc worker-1196" 0x00007feafc0b8ad3 in ?? ()
80 LWP 11969 "rpc worker-1196" 0x00007feafc0b8ad3 in ?? ()
81 LWP 11970 "rpc worker-1197" 0x00007feafc0b8ad3 in ?? ()
82 LWP 11971 "rpc worker-1197" 0x00007feafc0b8ad3 in ?? ()
83 LWP 11972 "rpc worker-1197" 0x00007feafc0b8ad3 in ?? ()
84 LWP 11973 "rpc worker-1197" 0x00007feafc0b8ad3 in ?? ()
85 LWP 11974 "rpc worker-1197" 0x00007feafc0b8ad3 in ?? ()
86 LWP 11975 "rpc worker-1197" 0x00007feafc0b8ad3 in ?? ()
87 LWP 11976 "rpc worker-1197" 0x00007feafc0b8ad3 in ?? ()
88 LWP 11977 "rpc worker-1197" 0x00007feafc0b8ad3 in ?? ()
89 LWP 11978 "rpc worker-1197" 0x00007feafc0b8ad3 in ?? ()
90 LWP 11979 "rpc worker-1197" 0x00007feafc0b8ad3 in ?? ()
91 LWP 11980 "rpc worker-1198" 0x00007feafc0b8ad3 in ?? ()
92 LWP 11981 "rpc worker-1198" 0x00007feafc0b8ad3 in ?? ()
93 LWP 11982 "rpc worker-1198" 0x00007feafc0b8ad3 in ?? ()
94 LWP 11983 "rpc worker-1198" 0x00007feafc0b8ad3 in ?? ()
95 LWP 11984 "rpc worker-1198" 0x00007feafc0b8ad3 in ?? ()
96 LWP 11985 "rpc worker-1198" 0x00007feafc0b8ad3 in ?? ()
97 LWP 11986 "rpc worker-1198" 0x00007feafc0b8ad3 in ?? ()
98 LWP 11987 "rpc worker-1198" 0x00007feafc0b8ad3 in ?? ()
99 LWP 11988 "rpc worker-1198" 0x00007feafc0b8ad3 in ?? ()
100 LWP 11989 "rpc worker-1198" 0x00007feafc0b8ad3 in ?? ()
101 LWP 11990 "rpc worker-1199" 0x00007feafc0b8ad3 in ?? ()
102 LWP 11991 "rpc worker-1199" 0x00007feafc0b8ad3 in ?? ()
103 LWP 11992 "rpc worker-1199" 0x00007feafc0b8ad3 in ?? ()
104 LWP 11993 "rpc worker-1199" 0x00007feafc0b8ad3 in ?? ()
105 LWP 11994 "rpc worker-1199" 0x00007feafc0b8ad3 in ?? ()
106 LWP 11995 "rpc worker-1199" 0x00007feafc0b8ad3 in ?? ()
107 LWP 11996 "rpc worker-1199" 0x00007feafc0b8ad3 in ?? ()
108 LWP 11997 "rpc worker-1199" 0x00007feafc0b8ad3 in ?? ()
109 LWP 11998 "rpc worker-1199" 0x00007feafc0b8ad3 in ?? ()
110 LWP 11999 "rpc worker-1199" 0x00007feafc0b8ad3 in ?? ()
111 LWP 12000 "rpc worker-1200" 0x00007feafc0b8ad3 in ?? ()
112 LWP 12001 "rpc worker-1200" 0x00007feafc0b8ad3 in ?? ()
113 LWP 12002 "rpc worker-1200" 0x00007feafc0b8ad3 in ?? ()
114 LWP 12003 "rpc worker-1200" 0x00007feafc0b8ad3 in ?? ()
115 LWP 12004 "rpc worker-1200" 0x00007feafc0b8ad3 in ?? ()
116 LWP 12005 "rpc worker-1200" 0x00007feafc0b8ad3 in ?? ()
117 LWP 12006 "diag-logger-120" 0x00007feafc0b8fb9 in ?? ()
118 LWP 12007 "result-tracker-" 0x00007feafc0b8fb9 in ?? ()
119 LWP 12008 "excess-log-dele" 0x00007feafc0b8fb9 in ?? ()
120 LWP 12009 "acceptor-12009" 0x00007feaf4e240c7 in ?? ()
121 LWP 12010 "heartbeat-12010" 0x00007feafc0b8fb9 in ?? ()
122 LWP 12011 "maintenance_sch" 0x00007feafc0b8fb9 in ?? ()
Thread 122 (LWP 12011):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000007e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007feab1184550 in ?? ()
#6 0x00000000000000fc in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 12010):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0xcf5ed54e759050c7 in ?? ()
#2 0x000000000000001f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007feab199c5b0 in ?? ()
#6 0x000000000000003e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 12009):
#0 0x00007feaf4e240c7 in ?? ()
#1 0x00006140000216a8 in ?? ()
#2 0x00007feab21c3d50 in ?? ()
#3 0x00007feab21c3d80 in ?? ()
#4 0x00007feab21c3e80 in ?? ()
#5 0x00007feab21c3d70 in ?? ()
#6 0x00007feab21c3de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007feab21c3f00 in ?? ()
#11 0x00007feab21c363c in ?? ()
#12 0x00000032b21c35b0 in ?? ()
#13 0x00007feab19c7000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 12008):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x00007feab29ddf00 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 12007):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x00007feab31f60c0 in ?? ()
#2 0x000000000000001f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007feab31f60b0 in ?? ()
#6 0x000000000000003e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 12006):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 12005):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007feab4225e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007feab4225e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007feafc0b8770 in ?? ()
#10 0x00007feab4225e70 in ?? ()
#11 0x00000ffd56844bd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 12004):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007feab4a3de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007feab4a3de70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007feafc0b8770 in ?? ()
#10 0x00007feab4a3de70 in ?? ()
#11 0x00000ffd56947bd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 12003):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 12002):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 12001):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 12000):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 11999):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 11998):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 11997):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 11996):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 11995):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 11994):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 11993):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 11992):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 11991):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 11990):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 11989):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 11988):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 11987):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 11986):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 11985):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 11984):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 11983):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 11982):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 11981):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 11980):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 11979):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 11978):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 11977):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 11976):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 11975):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 11974):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 11973):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 11972):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 11971):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 11970):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 11969):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 11968):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 11967):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 11966):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 11965):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007feac85f6e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007feac85f6e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 11964):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 11963):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 11962):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 11961):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 11960):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 11959):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 11958):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 11957):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 11956):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 11955):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 11954):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 11953):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 11952):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 11951):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 11950):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 11949):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 11948):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 11947):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 11946):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 11945):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007fead27d7e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fead27d7e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 11944):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 11943):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 11942):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 11941):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 11940):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 11939):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 11938):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 11937):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 11936):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 11935):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 11934):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 11933):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 11932):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 11931):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 11930):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 11929):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 11928):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 11927):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 11926):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 11925):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007feadc9afe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007feadc9afe70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 11924):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 11923):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 11922):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 11921):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 11920):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 11919):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 11918):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 11917):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 11916):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 11915):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 11914):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 11913):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 11912):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 11911):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 11910):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 11909):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 11908):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 11907):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 11906):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 11905):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x00007feae69c3c80 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007feae69c3c70 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 11904):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 11903):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x00007feae79c51c0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007feae79c51b0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 11902):
#0 0x00007feafc0b8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 11900):
#0 0x00007feaf4e22a47 in ?? ()
#1 0x00007feae89c82e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007feae89c82d0 in ?? ()
#4 0x00007feae89c84e0 in ?? ()
#5 0x00007feae89c8320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007feae89c83a0 in ?? ()
#8 0x00007feaf748225d in ?? ()
#9 0x3fb9646323655000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x00000000aa000000 in ?? ()
#14 0x00007fea00000000 in ?? ()
#15 0x41d9e134dfe3c369 in ?? ()
#16 0x00000ffddd131074 in ?? ()
#17 0x00007feae89c8380 in ?? ()
#18 0x00007feaf7486ba3 in ?? ()
#19 0x00007feae89c8350 in ?? ()
#20 0x3fb9646323655000 in ?? ()
#21 0x00000000e89c83a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb9646323655000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 11899):
#0 0x00007feaf4e22a47 in ?? ()
#1 0x00007feae91c92e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007feae91c92d0 in ?? ()
#4 0x00007feae91c94e0 in ?? ()
#5 0x00007feae91c9320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007feae91c93a0 in ?? ()
#8 0x00007feaf748225d in ?? ()
#9 0x3fb984b057c2f000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000043800000 in ?? ()
#14 0x00007fea00000000 in ?? ()
#15 0x41d9e134dfe3c366 in ?? ()
#16 0x00000ffddd231274 in ?? ()
#17 0x00007feae91c9380 in ?? ()
#18 0x00007feaf7486ba3 in ?? ()
#19 0x00007feae91c9350 in ?? ()
#20 0x3fb984b057c2f000 in ?? ()
#21 0x00000000e91c93a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb984b057c2f000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 11898):
#0 0x00007feaf4e22a47 in ?? ()
#1 0x00007feae99ca2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007feae99ca2d0 in ?? ()
#4 0x00007feae99ca4e0 in ?? ()
#5 0x00007feae99ca320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007feae99ca3a0 in ?? ()
#8 0x00007feaf748225d in ?? ()
#9 0x3fb95ec7a9b2f000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x00000000ff000000 in ?? ()
#14 0x00007fea00000000 in ?? ()
#15 0x41d9e134dfe3c367 in ?? ()
#16 0x00000ffddd331474 in ?? ()
#17 0x00007feae99ca380 in ?? ()
#18 0x00007feaf7486ba3 in ?? ()
#19 0x00007feae99ca350 in ?? ()
#20 0x3fb95ec7a9b2f000 in ?? ()
#21 0x00000000e99ca3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb95ec7a9b2f000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 11897):
#0 0x00007feaf4e22a47 in ?? ()
#1 0x00007feaeb5b32e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007feaeb5b32d0 in ?? ()
#4 0x00007feaeb5b34e0 in ?? ()
#5 0x00007feaeb5b3320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007feaeb5b33a0 in ?? ()
#8 0x00007feaf748225d in ?? ()
#9 0x3fb9620332f5f000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000b6800000 in ?? ()
#14 0x00007fea00000000 in ?? ()
#15 0x41d9e134dfe3c368 in ?? ()
#16 0x00000ffddd6ae674 in ?? ()
#17 0x00007feaeb5b3380 in ?? ()
#18 0x00007feaf7486ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 11894):
#0 0x00007feaf4e15cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007feaed5b7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 11893):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x00007feaecdb6080 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 11892):
#0 0x00007feafc0bc9e2 in ?? ()
#1 0x00007feaebdb2b60 in ?? ()
#2 0x00007feaebdb2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007feaebdb2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007feaebdb2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007feaebdb2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007feb019e0f0c in ?? ()
#12 0x00007feb019d07b9 in ?? ()
#13 0x00007feb019d4da6 in ?? ()
#14 0x00007feb019d97e0 in ?? ()
#15 0x00007feb019d9609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007feaf8ccf0be in ?? ()
#18 0x00007feafc0b26db in ?? ()
#19 0x00000ffddd7ae47c in ?? ()
#20 0x00007feaebdb2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 11886):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x00007feaee5b8c40 in ?? ()
#2 0x00000000000000a0 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007feaee5b8c30 in ?? ()
#6 0x0000000000000140 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 11885):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 11884):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 11883):
#0 0x00007feafc0b8fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007feaefdbc410 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 11882):
#0 0x00007feafc0bcd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 08:56:55.531469 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 11743
I20250113 08:56:55.543843 11869 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:56:55.809283 12447 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e3f389d6e3484a13af0780989a117f02)
I20250113 08:56:55.809547 12447 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:55.811545 12447 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), cf369533d8ad453f9d6c47cad3706dab (127.7.123.3:43245)
I20250113 08:56:55.821931 12100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "5f8ea89390544489a231c733997d2d9a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4303 } } ignore_live_leader: false dest_uuid: "cf369533d8ad453f9d6c47cad3706dab" is_pre_election: true
I20250113 08:56:55.846114 12451 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e3f389d6e3484a13af0780989a117f02)
I20250113 08:56:55.846436 12451 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:55.848220 12451 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333)
I20250113 08:56:55.857761 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "cf369533d8ad453f9d6c47cad3706dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 4303 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a" is_pre_election: true
I20250113 08:56:55.858103 11690 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf369533d8ad453f9d6c47cad3706dab in term 8.
I20250113 08:56:55.858798 12035 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab; no voters:
I20250113 08:56:55.859422 12451 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250113 08:56:55.859562 12451 raft_consensus.cc:491] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 8 FOLLOWER]: Starting leader election (detected failure of leader e3f389d6e3484a13af0780989a117f02)
I20250113 08:56:55.859692 12451 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:56:55.863119 12451 raft_consensus.cc:513] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:55.864013 12451 leader_election.cc:290] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 9 election: Requested vote from peers e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389), 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333)
I20250113 08:56:55.864769 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13658abfdf04a8c9fcc1f08c15ff92a" candidate_uuid: "cf369533d8ad453f9d6c47cad3706dab" candidate_term: 9 candidate_status { last_received { term: 8 index: 4303 } } ignore_live_leader: false dest_uuid: "5f8ea89390544489a231c733997d2d9a"
I20250113 08:56:55.864988 11690 raft_consensus.cc:3054] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 8 FOLLOWER]: Advancing to term 9
I20250113 08:56:55.867094 11690 raft_consensus.cc:2463] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf369533d8ad453f9d6c47cad3706dab in term 9.
I20250113 08:56:55.867771 12035 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a, cf369533d8ad453f9d6c47cad3706dab; no voters:
I20250113 08:56:55.868183 12451 raft_consensus.cc:2798] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 9 FOLLOWER]: Leader election won for term 9
I20250113 08:56:55.868732 12451 raft_consensus.cc:695] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 9 LEADER]: Becoming Leader. State: Replica: cf369533d8ad453f9d6c47cad3706dab, State: Running, Role: LEADER
I20250113 08:56:55.869707 12451 consensus_queue.cc:237] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4303, Committed index: 4303, Last appended: 8.4303, Last appended by leader: 4303, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } }
I20250113 08:56:55.874348 8937 catalog_manager.cc:5526] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab reported cstate change: term changed from 8 to 9, leader changed from e3f389d6e3484a13af0780989a117f02 (127.7.123.1) to cf369533d8ad453f9d6c47cad3706dab (127.7.123.3). New cstate: current_term: 9 leader_uuid: "cf369533d8ad453f9d6c47cad3706dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f389d6e3484a13af0780989a117f02" member_type: VOTER last_known_addr { host: "127.7.123.1" port: 38389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf369533d8ad453f9d6c47cad3706dab" member_type: VOTER last_known_addr { host: "127.7.123.3" port: 43245 } health_report { overall_health: HEALTHY } } }
I20250113 08:56:55.906394 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 11743
I20250113 08:56:55.907194 12100 raft_consensus.cc:1235] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [term 9 LEADER]: Rejecting Update request from peer e3f389d6e3484a13af0780989a117f02 for earlier term 8. Current term is 9. Ops: []
I20250113 08:56:55.909104 11690 raft_consensus.cc:1235] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 9 FOLLOWER]: Rejecting Update request from peer e3f389d6e3484a13af0780989a117f02 for earlier term 8. Current term is 9. Ops: []
W20250113 08:56:55.965651 12034 connection.cc:537] server connection from 127.7.123.1:38457 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:56:55.966229 11624 connection.cc:537] server connection from 127.7.123.1:51139 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:56:55.966535 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 11607
W20250113 08:56:55.966893 12448 negotiation.cc:337] Failed RPC negotiation. Trace:
0113 08:56:55.811821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.123.1:38389 (local address 127.7.123.2:60779)
0113 08:56:55.814704 (+ 2883us) negotiation.cc:107] Waiting for socket to connect
0113 08:56:55.814733 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0113 08:56:55.814999 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0113 08:56:55.908604 (+ 93605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0113 08:56:55.908610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0113 08:56:55.908663 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0113 08:56:55.909452 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0113 08:56:55.909468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0113 08:56:55.965832 (+ 56364us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.123.1:38389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2717,"thread_start_us":2563,"threads_started":1}
W20250113 08:56:55.966926 12452 negotiation.cc:337] Failed RPC negotiation. Trace:
0113 08:56:55.848482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.123.1:38389 (local address 127.7.123.3:37611)
0113 08:56:55.849140 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0113 08:56:55.849161 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0113 08:56:55.849294 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0113 08:56:55.965715 (+116421us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.123.1:38389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":393,"threads_started":1}
W20250113 08:56:55.967255 12036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.123.1:38389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:56:55.967253 11626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.123.1:38389: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:56:55.969403 11626 leader_election.cc:336] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389): Network error: Client connection negotiation failed: client connection to 127.7.123.1:38389: connect: Connection refused (error 111)
I20250113 08:56:55.969607 11626 leader_election.cc:304] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f8ea89390544489a231c733997d2d9a; no voters: cf369533d8ad453f9d6c47cad3706dab, e3f389d6e3484a13af0780989a117f02
W20250113 08:56:55.969702 12036 leader_election.cc:336] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389): Network error: Client connection negotiation failed: client connection to 127.7.123.1:38389: connect: Connection refused (error 111)
I20250113 08:56:55.970096 12447 raft_consensus.cc:2743] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250113 08:56:55.970158 12036 leader_election.cc:336] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389): Network error: Client connection negotiation failed: client connection to 127.7.123.1:38389: connect: Connection refused (error 111)
I20250113 08:56:55.974936 11731 generic_service.cc:196] Checking for leaks (request via RPC)
W20250113 08:56:55.984107 12036 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab -> Peer e3f389d6e3484a13af0780989a117f02 (127.7.123.1:38389): Couldn't send request to peer e3f389d6e3484a13af0780989a117f02. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.1:38389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:56:56.322243 11690 raft_consensus.cc:1270] T b13658abfdf04a8c9fcc1f08c15ff92a P 5f8ea89390544489a231c733997d2d9a [term 9 FOLLOWER]: Refusing update from remote peer cf369533d8ad453f9d6c47cad3706dab: Log matching property violated. Preceding OpId in replica: term: 8 index: 4303. Preceding OpId from leader: term: 9 index: 4304. (index mismatch)
I20250113 08:56:56.322875 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 11607
I20250113 08:56:56.325477 12451 consensus_queue.cc:1035] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8ea89390544489a231c733997d2d9a" member_type: VOTER last_known_addr { host: "127.7.123.2" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4304, Last known committed idx: 4303, Time since last communication: 0.000s
W20250113 08:56:56.382761 12035 connection.cc:537] client connection to 127.7.123.2:43333 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:56:56.382941 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 12017
W20250113 08:56:56.384384 12035 consensus_peers.cc:487] T b13658abfdf04a8c9fcc1f08c15ff92a P cf369533d8ad453f9d6c47cad3706dab -> Peer 5f8ea89390544489a231c733997d2d9a (127.7.123.2:43333): Couldn't send request to peer 5f8ea89390544489a231c733997d2d9a. Status: Network error: Client connection negotiation failed: client connection to 127.7.123.2:43333: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:56:56.389932 12140 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:56:56.741994 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 12017
I20250113 08:56:56.804803 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 11882
I20250113 08:56:56.812875 12004 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:56:56.912171 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 11882
I20250113 08:56:56.936429 7660 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu pid 8908
I20250113 08:56:56.943665 8969 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:56:57.114117 7660 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmZWr4w/build/asan/bin/kudu with pid 8908
2025-01-13T08:56:57Z chronyd exiting
I20250113 08:56:57.170398 7660 test_util.cc:181] -----------------------------------------------
I20250113 08:56:57.170586 7660 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskmZWr4w/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1736758448977324-7660-0
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/4, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 04-00 00-00 BE-BE BE-BE A0-24 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-24 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-27 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 04-00 00-00 03-00 00-00 01-BE BE-BE BE-BE BE-BE> (113655 ms)
[----------] 1 test from RollingRestartArgs/RollingRestartITest (113655 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 2 test suites ran. (168113 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/4, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 04-00 00-00 BE-BE BE-BE A0-24 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-24 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-27 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 04-00 00-00 03-00 00-00 01-BE BE-BE BE-BE BE-BE>
1 FAILED TEST
I20250113 08:56:57.172731 7660 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 6 messages since previous log ~31 seconds ago