Note: This is test shard 6 of 8.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RollingRestartArgs/RollingRestartITest
[ RUN ] RollingRestartArgs/RollingRestartITest.TestWorkloads/1
2025-06-25T03:42:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T03:42:57Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250625 03:42:57.348493 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.175.190:43735
--webserver_interface=127.12.175.190
--webserver_port=0
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.175.190:43735
--location_mapping_cmd=/tmp/dist-test-taskU8fiuI/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/location-assignment.state --map /L0:6
--master_client_location_assignment_enabled=false with env {}
W20250625 03:42:57.748225 12998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:42:57.748689 12998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:42:57.748837 12998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:42:57.757489 12998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 03:42:57.757639 12998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:42:57.757710 12998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 03:42:57.757794 12998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 03:42:57.768569 12998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/wal
--location_mapping_cmd=/tmp/dist-test-taskU8fiuI/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/location-assignment.state --map /L0:6
--ipki_ca_key_size=768
--master_addresses=127.12.175.190:43735
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.175.190:43735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.175.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:42:57.770783 12998 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:42:57.773268 12998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:42:57.785984 13003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:42:57.785916 13006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:42:57.786021 13004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:42:57.786998 12998 server_base.cc:1048] running on GCE node
I20250625 03:42:57.788053 12998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:42:57.789932 12998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:42:57.791167 12998 hybrid_clock.cc:648] HybridClock initialized: now 1750822977791102 us; error 56 us; skew 500 ppm
I20250625 03:42:57.791580 12998 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:42:57.794456 12998 webserver.cc:469] Webserver started at http://127.12.175.190:43571/ using document root <none> and password file <none>
I20250625 03:42:57.795409 12998 fs_manager.cc:362] Metadata directory not provided
I20250625 03:42:57.795527 12998 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:42:57.796048 12998 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 03:42:57.799046 12998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/data/instance:
uuid: "7349fb30bd4546798ab374350c9b0bfa"
format_stamp: "Formatted at 2025-06-25 03:42:57 on dist-test-slave-ttt3"
I20250625 03:42:57.800045 12998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/wal/instance:
uuid: "7349fb30bd4546798ab374350c9b0bfa"
format_stamp: "Formatted at 2025-06-25 03:42:57 on dist-test-slave-ttt3"
I20250625 03:42:57.806763 12998 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250625 03:42:57.811002 13012 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:42:57.812690 12998 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 03:42:57.812899 12998 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/wal
uuid: "7349fb30bd4546798ab374350c9b0bfa"
format_stamp: "Formatted at 2025-06-25 03:42:57 on dist-test-slave-ttt3"
I20250625 03:42:57.813179 12998 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:42:57.840495 12998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:42:57.841737 12998 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:42:57.842084 12998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:42:57.866972 12998 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.190:43735
I20250625 03:42:57.866982 13063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.190:43735 every 8 connection(s)
I20250625 03:42:57.869117 12998 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/data/info.pb
I20250625 03:42:57.874397 13064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 03:42:57.875500 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 12998
I20250625 03:42:57.875856 12990 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/wal/instance
I20250625 03:42:57.887619 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa: Bootstrap starting.
I20250625 03:42:57.891445 13064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa: Neither blocks nor log segments found. Creating new log.
I20250625 03:42:57.892925 13064 log.cc:826] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa: Log is configured to *not* fsync() on all Append() calls
I20250625 03:42:57.896283 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa: No bootstrap required, opened a new log
I20250625 03:42:57.902686 13064 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7349fb30bd4546798ab374350c9b0bfa" member_type: VOTER last_known_addr { host: "127.12.175.190" port: 43735 } }
I20250625 03:42:57.903167 13064 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 03:42:57.903322 13064 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7349fb30bd4546798ab374350c9b0bfa, State: Initialized, Role: FOLLOWER
I20250625 03:42:57.904214 13064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7349fb30bd4546798ab374350c9b0bfa" member_type: VOTER last_known_addr { host: "127.12.175.190" port: 43735 } }
I20250625 03:42:57.904534 13064 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 03:42:57.904686 13064 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 03:42:57.904930 13064 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [term 0 FOLLOWER]: Advancing to term 1
I20250625 03:42:57.907891 13064 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7349fb30bd4546798ab374350c9b0bfa" member_type: VOTER last_known_addr { host: "127.12.175.190" port: 43735 } }
I20250625 03:42:57.908699 13064 leader_election.cc:304] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7349fb30bd4546798ab374350c9b0bfa; no voters:
I20250625 03:42:57.909497 13064 leader_election.cc:290] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 03:42:57.909718 13069 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [term 1 FOLLOWER]: Leader election won for term 1
I20250625 03:42:57.910533 13069 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [term 1 LEADER]: Becoming Leader. State: Replica: 7349fb30bd4546798ab374350c9b0bfa, State: Running, Role: LEADER
I20250625 03:42:57.911206 13069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7349fb30bd4546798ab374350c9b0bfa" member_type: VOTER last_known_addr { host: "127.12.175.190" port: 43735 } }
I20250625 03:42:57.912328 13064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [sys.catalog]: configured and running, proceeding with master startup.
I20250625 03:42:57.914994 13071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7349fb30bd4546798ab374350c9b0bfa. Latest consensus state: current_term: 1 leader_uuid: "7349fb30bd4546798ab374350c9b0bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7349fb30bd4546798ab374350c9b0bfa" member_type: VOTER last_known_addr { host: "127.12.175.190" port: 43735 } } }
I20250625 03:42:57.915486 13071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [sys.catalog]: This master's current role is: LEADER
I20250625 03:42:57.918176 13070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7349fb30bd4546798ab374350c9b0bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7349fb30bd4546798ab374350c9b0bfa" member_type: VOTER last_known_addr { host: "127.12.175.190" port: 43735 } } }
I20250625 03:42:57.918510 13070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa [sys.catalog]: This master's current role is: LEADER
I20250625 03:42:57.920892 13074 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 03:42:57.927048 13074 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 03:42:57.938019 13074 catalog_manager.cc:1349] Generated new cluster ID: 12059e825d0b4b4589905c35743e7d23
I20250625 03:42:57.938201 13074 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 03:42:57.962064 13074 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 03:42:57.963678 13074 catalog_manager.cc:1506] Loading token signing keys...
I20250625 03:42:57.985826 13074 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7349fb30bd4546798ab374350c9b0bfa: Generated new TSK 0
I20250625 03:42:57.987210 13074 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 03:42:57.994660 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.129:0
--local_ip_for_outbound_sockets=127.12.175.129
--webserver_interface=127.12.175.129
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250625 03:42:58.395617 13088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:42:58.396164 13088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:42:58.396334 13088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:42:58.405597 13088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:42:58.405978 13088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.129
I20250625 03:42:58.417369 13088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.175.129
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.129
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:42:58.419624 13088 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:42:58.421962 13088 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:42:58.437906 13094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:42:58.437949 13093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:42:58.438001 13096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:42:58.439491 13088 server_base.cc:1048] running on GCE node
I20250625 03:42:58.440482 13088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:42:58.441983 13088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:42:58.443305 13088 hybrid_clock.cc:648] HybridClock initialized: now 1750822978443226 us; error 60 us; skew 500 ppm
I20250625 03:42:58.443980 13088 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:42:58.447284 13088 webserver.cc:469] Webserver started at http://127.12.175.129:34437/ using document root <none> and password file <none>
I20250625 03:42:58.448462 13088 fs_manager.cc:362] Metadata directory not provided
I20250625 03:42:58.448596 13088 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:42:58.449035 13088 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 03:42:58.452224 13088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data/instance:
uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f"
format_stamp: "Formatted at 2025-06-25 03:42:58 on dist-test-slave-ttt3"
I20250625 03:42:58.453145 13088 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal/instance:
uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f"
format_stamp: "Formatted at 2025-06-25 03:42:58 on dist-test-slave-ttt3"
I20250625 03:42:58.459565 13088 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250625 03:42:58.463567 13102 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:42:58.465148 13088 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 03:42:58.465332 13088 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal
uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f"
format_stamp: "Formatted at 2025-06-25 03:42:58 on dist-test-slave-ttt3"
I20250625 03:42:58.465576 13088 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:42:58.485890 13088 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:42:58.487054 13088 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:42:58.487372 13088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:42:58.488577 13088 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:42:58.490876 13088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 03:42:58.490991 13088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 03:42:58.491127 13088 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 03:42:58.491210 13088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 03:42:58.529086 13088 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.129:45233
I20250625 03:42:58.529162 13214 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.129:45233 every 8 connection(s)
I20250625 03:42:58.530910 13088 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data/info.pb
I20250625 03:42:58.532240 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 13088
I20250625 03:42:58.532513 12990 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal/instance
I20250625 03:42:58.536948 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.130:0
--local_ip_for_outbound_sockets=127.12.175.130
--webserver_interface=127.12.175.130
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250625 03:42:58.547349 13215 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:42:58.547858 13215 heartbeater.cc:461] Registering TS with master...
I20250625 03:42:58.549160 13215 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:42:58.609911 13029 ts_manager.cc:194] Registered new tserver with Master: 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233)
I20250625 03:42:58.612870 13029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.129:35989
W20250625 03:42:58.941265 13219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:42:58.941721 13219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:42:58.941854 13219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:42:58.950794 13219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:42:58.951081 13219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.130
I20250625 03:42:58.961694 13219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.175.130
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.130
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:42:58.964010 13219 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:42:58.966804 13219 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:42:58.985343 13226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:42:58.989970 13225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:42:58.995009 13219 server_base.cc:1048] running on GCE node
W20250625 03:42:58.998786 13228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:42:59.000013 13219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:42:59.001621 13219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:42:59.002987 13219 hybrid_clock.cc:648] HybridClock initialized: now 1750822979002881 us; error 96 us; skew 500 ppm
I20250625 03:42:59.003587 13219 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:42:59.011200 13219 webserver.cc:469] Webserver started at http://127.12.175.130:37683/ using document root <none> and password file <none>
I20250625 03:42:59.012347 13219 fs_manager.cc:362] Metadata directory not provided
I20250625 03:42:59.012496 13219 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:42:59.012929 13219 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 03:42:59.016216 13219 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data/instance:
uuid: "975eeb611762460bb2f09f5b4b71e3d4"
format_stamp: "Formatted at 2025-06-25 03:42:59 on dist-test-slave-ttt3"
I20250625 03:42:59.017226 13219 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal/instance:
uuid: "975eeb611762460bb2f09f5b4b71e3d4"
format_stamp: "Formatted at 2025-06-25 03:42:59 on dist-test-slave-ttt3"
I20250625 03:42:59.024533 13219 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20250625 03:42:59.030297 13234 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:42:59.032574 13219 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250625 03:42:59.032814 13219 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal
uuid: "975eeb611762460bb2f09f5b4b71e3d4"
format_stamp: "Formatted at 2025-06-25 03:42:59 on dist-test-slave-ttt3"
I20250625 03:42:59.033089 13219 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:42:59.061558 13219 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:42:59.063022 13219 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:42:59.063467 13219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:42:59.065219 13219 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:42:59.068523 13219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 03:42:59.068666 13219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 03:42:59.068819 13219 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 03:42:59.068907 13219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 03:42:59.109324 13219 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.130:34057
I20250625 03:42:59.109398 13346 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.130:34057 every 8 connection(s)
I20250625 03:42:59.111070 13219 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data/info.pb
I20250625 03:42:59.116338 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 13219
I20250625 03:42:59.116595 12990 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal/instance
I20250625 03:42:59.121271 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.131:0
--local_ip_for_outbound_sockets=127.12.175.131
--webserver_interface=127.12.175.131
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250625 03:42:59.128741 13347 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:42:59.129220 13347 heartbeater.cc:461] Registering TS with master...
I20250625 03:42:59.130457 13347 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:42:59.181175 13029 ts_manager.cc:194] Registered new tserver with Master: 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:42:59.182353 13029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.130:57805
W20250625 03:42:59.507877 13351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:42:59.508337 13351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:42:59.508489 13351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:42:59.517694 13351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:42:59.517973 13351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.131
I20250625 03:42:59.529175 13351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.175.131
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.131
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:42:59.531616 13351 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:42:59.534523 13351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:42:59.548585 13360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:42:59.548642 13358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:42:59.548691 13357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:42:59.549527 13351 server_base.cc:1048] running on GCE node
I20250625 03:42:59.550249 13351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:42:59.551398 13351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:42:59.552627 13351 hybrid_clock.cc:648] HybridClock initialized: now 1750822979552584 us; error 41 us; skew 500 ppm
I20250625 03:42:59.552987 13351 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:42:59.555737 13351 webserver.cc:469] Webserver started at http://127.12.175.131:38451/ using document root <none> and password file <none>
I20250625 03:42:59.556690 13351 fs_manager.cc:362] Metadata directory not provided
I20250625 03:42:59.556813 13351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:42:59.557229 13351 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 03:42:59.560222 13351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data/instance:
uuid: "513d19d66cd54765940762233636fd31"
format_stamp: "Formatted at 2025-06-25 03:42:59 on dist-test-slave-ttt3"
I20250625 03:42:59.561131 13351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal/instance:
uuid: "513d19d66cd54765940762233636fd31"
format_stamp: "Formatted at 2025-06-25 03:42:59 on dist-test-slave-ttt3"
I20250625 03:42:59.567727 13351 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250625 03:42:59.571913 13366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:42:59.573665 13351 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 03:42:59.573887 13351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal
uuid: "513d19d66cd54765940762233636fd31"
format_stamp: "Formatted at 2025-06-25 03:42:59 on dist-test-slave-ttt3"
I20250625 03:42:59.574147 13351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:42:59.608614 13351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:42:59.609709 13351 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:42:59.610029 13351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:42:59.611275 13351 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:42:59.613389 13351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 03:42:59.613513 13351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 03:42:59.613636 13351 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 03:42:59.613714 13351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 03:42:59.617163 13215 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
I20250625 03:42:59.652369 13351 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.131:40037
I20250625 03:42:59.652472 13478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.131:40037 every 8 connection(s)
I20250625 03:42:59.654089 13351 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data/info.pb
I20250625 03:42:59.658656 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 13351
I20250625 03:42:59.658890 12990 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal/instance
I20250625 03:42:59.662895 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.132:0
--local_ip_for_outbound_sockets=127.12.175.132
--webserver_interface=127.12.175.132
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250625 03:42:59.671389 13479 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:42:59.671902 13479 heartbeater.cc:461] Registering TS with master...
I20250625 03:42:59.672973 13479 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:42:59.723037 13029 ts_manager.cc:194] Registered new tserver with Master: 513d19d66cd54765940762233636fd31 (127.12.175.131:40037)
I20250625 03:42:59.724303 13029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.131:33859
W20250625 03:43:00.052945 13483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:43:00.053401 13483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:43:00.053567 13483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:43:00.062497 13483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:43:00.062757 13483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.132
I20250625 03:43:00.073750 13483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.12.175.132
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.132
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:43:00.075917 13483 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:43:00.078348 13483 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:43:00.093202 13490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:00.093122 13489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:00.093147 13492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:43:00.094240 13483 server_base.cc:1048] running on GCE node
I20250625 03:43:00.095017 13483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:43:00.096333 13483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:43:00.097574 13483 hybrid_clock.cc:648] HybridClock initialized: now 1750822980097519 us; error 51 us; skew 500 ppm
I20250625 03:43:00.097977 13483 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:43:00.101189 13483 webserver.cc:469] Webserver started at http://127.12.175.132:39777/ using document root <none> and password file <none>
I20250625 03:43:00.102151 13483 fs_manager.cc:362] Metadata directory not provided
I20250625 03:43:00.102272 13483 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:43:00.102689 13483 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 03:43:00.105792 13483 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data/instance:
uuid: "d275425c3ef443f88abb12d24f4c2f2f"
format_stamp: "Formatted at 2025-06-25 03:43:00 on dist-test-slave-ttt3"
I20250625 03:43:00.106770 13483 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal/instance:
uuid: "d275425c3ef443f88abb12d24f4c2f2f"
format_stamp: "Formatted at 2025-06-25 03:43:00 on dist-test-slave-ttt3"
I20250625 03:43:00.114861 13483 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20250625 03:43:00.119907 13498 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:43:00.121649 13483 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250625 03:43:00.121876 13483 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal
uuid: "d275425c3ef443f88abb12d24f4c2f2f"
format_stamp: "Formatted at 2025-06-25 03:43:00 on dist-test-slave-ttt3"
I20250625 03:43:00.122156 13483 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:43:00.138720 13483 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:43:00.139982 13483 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:43:00.140316 13483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:43:00.141546 13483 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:43:00.143623 13483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 03:43:00.143774 13483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250625 03:43:00.143916 13483 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 03:43:00.144001 13483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 03:43:00.184059 13483 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.132:42473
I20250625 03:43:00.184121 13610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.132:42473 every 8 connection(s)
I20250625 03:43:00.185789 13347 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
I20250625 03:43:00.185865 13483 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data/info.pb
I20250625 03:43:00.191371 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 13483
I20250625 03:43:00.191622 12990 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal/instance
I20250625 03:43:00.195639 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.133:0
--local_ip_for_outbound_sockets=127.12.175.133
--webserver_interface=127.12.175.133
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250625 03:43:00.202620 13611 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:43:00.203127 13611 heartbeater.cc:461] Registering TS with master...
I20250625 03:43:00.204468 13611 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:00.256145 13029 ts_manager.cc:194] Registered new tserver with Master: d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:00.257401 13029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.132:57727
W20250625 03:43:00.588909 13615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:43:00.589339 13615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:43:00.589493 13615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:43:00.598342 13615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:43:00.598627 13615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.133
I20250625 03:43:00.609714 13615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.12.175.133
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.133
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:43:00.611850 13615 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:43:00.614104 13615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:43:00.628145 13622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:00.628176 13621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:00.628207 13624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:43:00.629562 13615 server_base.cc:1048] running on GCE node
I20250625 03:43:00.630247 13615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:43:00.631438 13615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:43:00.632658 13615 hybrid_clock.cc:648] HybridClock initialized: now 1750822980632575 us; error 73 us; skew 500 ppm
I20250625 03:43:00.633003 13615 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:43:00.635650 13615 webserver.cc:469] Webserver started at http://127.12.175.133:38511/ using document root <none> and password file <none>
I20250625 03:43:00.636646 13615 fs_manager.cc:362] Metadata directory not provided
I20250625 03:43:00.636754 13615 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:43:00.637132 13615 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 03:43:00.640249 13615 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data/instance:
uuid: "506230f3ca974bf796583a6be750e38f"
format_stamp: "Formatted at 2025-06-25 03:43:00 on dist-test-slave-ttt3"
I20250625 03:43:00.641170 13615 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal/instance:
uuid: "506230f3ca974bf796583a6be750e38f"
format_stamp: "Formatted at 2025-06-25 03:43:00 on dist-test-slave-ttt3"
I20250625 03:43:00.647338 13615 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250625 03:43:00.651454 13630 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:43:00.653384 13615 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20250625 03:43:00.653589 13615 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal
uuid: "506230f3ca974bf796583a6be750e38f"
format_stamp: "Formatted at 2025-06-25 03:43:00 on dist-test-slave-ttt3"
I20250625 03:43:00.653864 13615 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:43:00.691255 13615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:43:00.692385 13615 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:43:00.692662 13615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:43:00.693776 13615 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:43:00.695652 13615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 03:43:00.695799 13615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250625 03:43:00.695897 13615 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 03:43:00.695957 13615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 03:43:00.728470 13479 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
I20250625 03:43:00.733490 13615 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.133:36385
I20250625 03:43:00.733548 13742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.133:36385 every 8 connection(s)
I20250625 03:43:00.735180 13615 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data/info.pb
I20250625 03:43:00.743348 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 13615
I20250625 03:43:00.743593 12990 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal/instance
I20250625 03:43:00.747642 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.134:0
--local_ip_for_outbound_sockets=127.12.175.134
--webserver_interface=127.12.175.134
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250625 03:43:00.751399 13743 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:43:00.751864 13743 heartbeater.cc:461] Registering TS with master...
I20250625 03:43:00.752923 13743 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:00.827522 13029 ts_manager.cc:194] Registered new tserver with Master: 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385)
I20250625 03:43:00.828821 13029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.133:34463
W20250625 03:43:01.164471 13747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:43:01.164965 13747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:43:01.165134 13747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:43:01.174011 13747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:43:01.174269 13747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.134
I20250625 03:43:01.186053 13747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.12.175.134
--webserver_port=0
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.134
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:43:01.188582 13747 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:43:01.191520 13747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:43:01.206143 13754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:01.206166 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:01.206257 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:43:01.207142 13747 server_base.cc:1048] running on GCE node
I20250625 03:43:01.207934 13747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:43:01.209136 13747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:43:01.210388 13747 hybrid_clock.cc:648] HybridClock initialized: now 1750822981210337 us; error 42 us; skew 500 ppm
I20250625 03:43:01.210757 13747 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:43:01.213862 13747 webserver.cc:469] Webserver started at http://127.12.175.134:36347/ using document root <none> and password file <none>
I20250625 03:43:01.214882 13747 fs_manager.cc:362] Metadata directory not provided
I20250625 03:43:01.215039 13747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:43:01.215487 13747 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 03:43:01.218834 13747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data/instance:
uuid: "c24f9318ff0c418292d80906d5f0fd95"
format_stamp: "Formatted at 2025-06-25 03:43:01 on dist-test-slave-ttt3"
I20250625 03:43:01.219990 13747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal/instance:
uuid: "c24f9318ff0c418292d80906d5f0fd95"
format_stamp: "Formatted at 2025-06-25 03:43:01 on dist-test-slave-ttt3"
I20250625 03:43:01.227355 13747 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250625 03:43:01.231899 13762 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:43:01.233512 13747 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250625 03:43:01.233727 13747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal
uuid: "c24f9318ff0c418292d80906d5f0fd95"
format_stamp: "Formatted at 2025-06-25 03:43:01 on dist-test-slave-ttt3"
I20250625 03:43:01.234004 13747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:43:01.261583 13611 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
I20250625 03:43:01.262904 13747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:43:01.264353 13747 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:43:01.264667 13747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:43:01.265842 13747 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:43:01.268110 13747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 03:43:01.268225 13747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 03:43:01.268329 13747 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 03:43:01.268397 13747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 03:43:01.308925 13747 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.134:40555
I20250625 03:43:01.309021 13874 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.134:40555 every 8 connection(s)
I20250625 03:43:01.310721 13747 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data/info.pb
I20250625 03:43:01.319171 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 13747
I20250625 03:43:01.319406 12990 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal/instance
I20250625 03:43:01.327945 13875 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:43:01.328464 13875 heartbeater.cc:461] Registering TS with master...
I20250625 03:43:01.329638 13875 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:01.385810 13029 ts_manager.cc:194] Registered new tserver with Master: c24f9318ff0c418292d80906d5f0fd95 (127.12.175.134:40555)
I20250625 03:43:01.386907 13029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.134:45703
I20250625 03:43:01.397802 12990 external_mini_cluster.cc:934] 6 TS(s) registered with all masters
I20250625 03:43:01.423054 12990 test_util.cc:276] Using random seed: 1038045083
I20250625 03:43:01.448290 13029 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54374:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 03:43:01.494937 13414 tablet_service.cc:1468] Processing CreateTablet for tablet 9126b7f95c9d434cb308581a7ce5b908 (DEFAULT_TABLE table=test-workload [id=7880011af54640e4bcc5a41f10f3d23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 03:43:01.500921 13414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9126b7f95c9d434cb308581a7ce5b908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 03:43:01.501735 13150 tablet_service.cc:1468] Processing CreateTablet for tablet 9126b7f95c9d434cb308581a7ce5b908 (DEFAULT_TABLE table=test-workload [id=7880011af54640e4bcc5a41f10f3d23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 03:43:01.501967 13282 tablet_service.cc:1468] Processing CreateTablet for tablet 9126b7f95c9d434cb308581a7ce5b908 (DEFAULT_TABLE table=test-workload [id=7880011af54640e4bcc5a41f10f3d23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 03:43:01.501976 13678 tablet_service.cc:1468] Processing CreateTablet for tablet 9126b7f95c9d434cb308581a7ce5b908 (DEFAULT_TABLE table=test-workload [id=7880011af54640e4bcc5a41f10f3d23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 03:43:01.503450 13282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9126b7f95c9d434cb308581a7ce5b908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 03:43:01.503453 13150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9126b7f95c9d434cb308581a7ce5b908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 03:43:01.503502 13678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9126b7f95c9d434cb308581a7ce5b908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 03:43:01.522545 13905 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Bootstrap starting.
I20250625 03:43:01.533749 13905 tablet_bootstrap.cc:654] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Neither blocks nor log segments found. Creating new log.
I20250625 03:43:01.533326 13546 tablet_service.cc:1468] Processing CreateTablet for tablet 9126b7f95c9d434cb308581a7ce5b908 (DEFAULT_TABLE table=test-workload [id=7880011af54640e4bcc5a41f10f3d23a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 03:43:01.535921 13904 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Bootstrap starting.
I20250625 03:43:01.536023 13546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9126b7f95c9d434cb308581a7ce5b908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 03:43:01.541940 13904 tablet_bootstrap.cc:654] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Neither blocks nor log segments found. Creating new log.
I20250625 03:43:01.544014 13904 log.cc:826] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Log is configured to *not* fsync() on all Append() calls
I20250625 03:43:01.544701 13907 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap starting.
I20250625 03:43:01.545820 13905 log.cc:826] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Log is configured to *not* fsync() on all Append() calls
I20250625 03:43:01.546164 13906 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Bootstrap starting.
I20250625 03:43:01.551213 13906 tablet_bootstrap.cc:654] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Neither blocks nor log segments found. Creating new log.
I20250625 03:43:01.553099 13906 log.cc:826] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Log is configured to *not* fsync() on all Append() calls
I20250625 03:43:01.555115 13904 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: No bootstrap required, opened a new log
I20250625 03:43:01.555629 13904 ts_tablet_manager.cc:1397] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.010s
I20250625 03:43:01.556064 13907 tablet_bootstrap.cc:654] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Neither blocks nor log segments found. Creating new log.
I20250625 03:43:01.556998 13905 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: No bootstrap required, opened a new log
I20250625 03:43:01.557374 13905 ts_tablet_manager.cc:1397] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.009s
I20250625 03:43:01.557750 13907 log.cc:826] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Log is configured to *not* fsync() on all Append() calls
I20250625 03:43:01.567595 13904 raft_consensus.cc:357] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.568408 13904 raft_consensus.cc:383] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 03:43:01.568579 13904 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513d19d66cd54765940762233636fd31, State: Initialized, Role: FOLLOWER
I20250625 03:43:01.569779 13904 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.572450 13906 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: No bootstrap required, opened a new log
I20250625 03:43:01.572890 13906 ts_tablet_manager.cc:1397] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.000s
I20250625 03:43:01.582222 13911 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Bootstrap starting.
I20250625 03:43:01.584872 13905 raft_consensus.cc:357] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.585744 13905 raft_consensus.cc:383] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 03:43:01.585959 13905 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975eeb611762460bb2f09f5b4b71e3d4, State: Initialized, Role: FOLLOWER
I20250625 03:43:01.586838 13906 raft_consensus.cc:357] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.587332 13906 raft_consensus.cc:383] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 03:43:01.587478 13906 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1b4dfaea74515a9e5aab1fbc0870f, State: Initialized, Role: FOLLOWER
I20250625 03:43:01.588102 13905 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.588516 13907 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: No bootstrap required, opened a new log
I20250625 03:43:01.588343 13906 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.588938 13907 ts_tablet_manager.cc:1397] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.021s
I20250625 03:43:01.589012 13904 ts_tablet_manager.cc:1428] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Time spent starting tablet: real 0.033s user 0.014s sys 0.018s
I20250625 03:43:01.590878 13906 ts_tablet_manager.cc:1428] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Time spent starting tablet: real 0.018s user 0.013s sys 0.000s
I20250625 03:43:01.593703 13911 tablet_bootstrap.cc:654] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Neither blocks nor log segments found. Creating new log.
I20250625 03:43:01.595532 13905 ts_tablet_manager.cc:1428] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Time spent starting tablet: real 0.038s user 0.013s sys 0.005s
I20250625 03:43:01.595600 13911 log.cc:826] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Log is configured to *not* fsync() on all Append() calls
I20250625 03:43:01.598940 13913 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 03:43:01.599268 13913 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.600229 13907 raft_consensus.cc:357] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.600783 13907 raft_consensus.cc:383] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 03:43:01.600951 13907 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506230f3ca974bf796583a6be750e38f, State: Initialized, Role: FOLLOWER
I20250625 03:43:01.601898 13907 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.608204 13911 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: No bootstrap required, opened a new log
I20250625 03:43:01.609086 13911 ts_tablet_manager.cc:1397] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.000s
I20250625 03:43:01.614547 13913 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
W20250625 03:43:01.615259 13348 tablet.cc:2378] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 03:43:01.619287 13743 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
I20250625 03:43:01.620466 13907 ts_tablet_manager.cc:1428] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Time spent starting tablet: real 0.031s user 0.017s sys 0.011s
I20250625 03:43:01.633095 13911 raft_consensus.cc:357] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.634435 13911 raft_consensus.cc:383] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 03:43:01.634657 13911 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d275425c3ef443f88abb12d24f4c2f2f, State: Initialized, Role: FOLLOWER
I20250625 03:43:01.634981 13917 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 03:43:01.635505 13302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:43:01.635615 13911 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.636271 13302 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 0.
I20250625 03:43:01.635854 13698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:43:01.637120 13917 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.640028 13698 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 0.
I20250625 03:43:01.640223 13911 ts_tablet_manager.cc:1428] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Time spent starting tablet: real 0.030s user 0.012s sys 0.005s
I20250625 03:43:01.641745 13369 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4; no voters:
I20250625 03:43:01.642414 13913 raft_consensus.cc:2802] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 03:43:01.642603 13913 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 03:43:01.642719 13913 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 0 FOLLOWER]: Advancing to term 1
I20250625 03:43:01.642940 13170 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:43:01.643519 13170 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 0.
I20250625 03:43:01.646191 13917 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:01.647565 13913 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.649066 13170 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f"
I20250625 03:43:01.649300 13170 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 0 FOLLOWER]: Advancing to term 1
I20250625 03:43:01.650127 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:43:01.650799 13566 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 0.
I20250625 03:43:01.653278 13170 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 1.
I20250625 03:43:01.654709 13913 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 1 election: Requested vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:01.655406 13302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4"
I20250625 03:43:01.655719 13302 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 0 FOLLOWER]: Advancing to term 1
I20250625 03:43:01.655928 13170 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "506230f3ca974bf796583a6be750e38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:43:01.656361 13170 raft_consensus.cc:2391] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 506230f3ca974bf796583a6be750e38f in current term 1: Already voted for candidate 513d19d66cd54765940762233636fd31 in this term.
W20250625 03:43:01.657799 13480 tablet.cc:2378] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 03:43:01.659557 13302 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 1.
I20250625 03:43:01.660121 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f"
I20250625 03:43:01.660356 13566 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 0 FOLLOWER]: Advancing to term 1
I20250625 03:43:01.664466 13370 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f; no voters:
I20250625 03:43:01.664943 13566 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 1.
I20250625 03:43:01.665040 13913 raft_consensus.cc:2802] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 03:43:01.665285 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "506230f3ca974bf796583a6be750e38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:43:01.667277 13913 raft_consensus.cc:695] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 1 LEADER]: Becoming Leader. State: Replica: 513d19d66cd54765940762233636fd31, State: Running, Role: LEADER
I20250625 03:43:01.667941 13698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f"
I20250625 03:43:01.668012 13913 consensus_queue.cc:237] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:01.668207 13698 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 0 FOLLOWER]: Advancing to term 1
I20250625 03:43:01.671773 13302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "506230f3ca974bf796583a6be750e38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:43:01.672349 13302 raft_consensus.cc:2391] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 506230f3ca974bf796583a6be750e38f in current term 1: Already voted for candidate 513d19d66cd54765940762233636fd31 in this term.
I20250625 03:43:01.674343 13698 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 1.
I20250625 03:43:01.676342 13632 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 506230f3ca974bf796583a6be750e38f; no voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:43:01.676951 13917 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 03:43:01.678515 13029 catalog_manager.cc:5582] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 513d19d66cd54765940762233636fd31 (127.12.175.131). New cstate: current_term: 1 leader_uuid: "513d19d66cd54765940762233636fd31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20250625 03:43:01.679492 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "506230f3ca974bf796583a6be750e38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:43:01.679948 13566 raft_consensus.cc:2391] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 506230f3ca974bf796583a6be750e38f in current term 1: Already voted for candidate 513d19d66cd54765940762233636fd31 in this term.
W20250625 03:43:01.689039 13612 tablet.cc:2378] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 03:43:01.718844 12990 maintenance_mode-itest.cc:745] Restarting batch of 2 tservers: 975eeb611762460bb2f09f5b4b71e3d4,9cb1b4dfaea74515a9e5aab1fbc0870f
W20250625 03:43:01.737886 13744 tablet.cc:2378] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 03:43:01.772194 13913 consensus_queue.cc:1035] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 03:43:01.783179 13913 consensus_queue.cc:1035] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 03:43:01.785774 13216 tablet.cc:2378] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 03:43:01.791445 13926 consensus_queue.cc:1035] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 03:43:01.792110 13926 consensus_queue.cc:1035] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 03:43:01.908515 13394 tablet_service.cc:2912] Rejecting scan request for tablet 9126b7f95c9d434cb308581a7ce5b908: Uninitialized: clean time has not yet been initialized
W20250625 03:43:01.908541 13392 tablet_service.cc:2912] Rejecting scan request for tablet 9126b7f95c9d434cb308581a7ce5b908: Uninitialized: clean time has not yet been initialized
W20250625 03:43:01.920548 13394 tablet_service.cc:2912] Rejecting scan request for tablet 9126b7f95c9d434cb308581a7ce5b908: Uninitialized: clean time has not yet been initialized
W20250625 03:43:01.973129 13367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.129:45233, user_credentials={real_user=slave}} blocked reactor thread for 70025us
I20250625 03:43:01.993243 13943 mvcc.cc:204] Tried to move back new op lower bound from 7171370933816020992 to 7171370932920348672. Current Snapshot: MvccSnapshot[applied={T|T < 7171370933816020992 or (T in {7171370933816020992})}]
I20250625 03:43:02.390470 13875 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
I20250625 03:43:03.197965 13023 ts_manager.cc:295] Set tserver state for 975eeb611762460bb2f09f5b4b71e3d4 to MAINTENANCE_MODE
I20250625 03:43:03.202387 13020 ts_manager.cc:295] Set tserver state for 9cb1b4dfaea74515a9e5aab1fbc0870f to MAINTENANCE_MODE
I20250625 03:43:04.633962 13282 tablet_service.cc:1423] Tablet server 975eeb611762460bb2f09f5b4b71e3d4 set to quiescing
I20250625 03:43:04.634317 13282 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:06.350006 13150 tablet_service.cc:1423] Tablet server 9cb1b4dfaea74515a9e5aab1fbc0870f set to quiescing
I20250625 03:43:06.350508 13150 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:07.148746 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 13219
I20250625 03:43:07.182926 13342 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 03:43:07.497574 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 13219
W20250625 03:43:07.594105 13370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.175.130:34057 (error 108)
W20250625 03:43:07.606494 14037 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 03:43:07.600882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.175.130:34057 (local address 127.12.175.131:60269)
0625 03:43:07.604803 (+ 3921us) negotiation.cc:107] Waiting for socket to connect
0625 03:43:07.605093 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":3802,"thread_start_us":3743,"threads_started":1}
W20250625 03:43:07.607257 13370 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): Couldn't send request to peer 975eeb611762460bb2f09f5b4b71e3d4. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20250625 03:43:07.608201 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.130:34057
--local_ip_for_outbound_sockets=127.12.175.130
--tserver_master_addrs=127.12.175.190:43735
--webserver_port=37683
--webserver_interface=127.12.175.130
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250625 03:43:08.102382 13370 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): Couldn't send request to peer 975eeb611762460bb2f09f5b4b71e3d4. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250625 03:43:08.557024 14038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:43:08.557696 14038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:43:08.557901 14038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:43:08.579584 14038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:43:08.580858 14038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.130
W20250625 03:43:08.595824 13370 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): Couldn't send request to peer 975eeb611762460bb2f09f5b4b71e3d4. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250625 03:43:08.598613 14038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.130:34057
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.175.130
--webserver_port=37683
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.130
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:43:08.601709 14038 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:43:08.605266 14038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:43:08.638706 14049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:08.640774 14050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:08.643415 14052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:43:08.647325 14038 server_base.cc:1048] running on GCE node
I20250625 03:43:08.648499 14038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:43:08.652645 14038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:43:08.662145 14038 hybrid_clock.cc:648] HybridClock initialized: now 1750822988662035 us; error 74 us; skew 500 ppm
I20250625 03:43:08.663210 14038 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:43:08.686406 14038 webserver.cc:469] Webserver started at http://127.12.175.130:37683/ using document root <none> and password file <none>
I20250625 03:43:08.690747 14038 fs_manager.cc:362] Metadata directory not provided
I20250625 03:43:08.691064 14038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:43:08.740061 14038 fs_manager.cc:714] Time spent opening directory manager: real 0.044s user 0.009s sys 0.024s
I20250625 03:43:08.773764 14060 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:43:08.784071 14038 fs_manager.cc:730] Time spent opening block manager: real 0.039s user 0.009s sys 0.011s
I20250625 03:43:08.784924 14038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal
uuid: "975eeb611762460bb2f09f5b4b71e3d4"
format_stamp: "Formatted at 2025-06-25 03:42:59 on dist-test-slave-ttt3"
I20250625 03:43:08.786495 14038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:43:08.851945 14038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:43:08.856155 14038 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:43:08.856673 14038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:43:08.871387 14038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:43:08.951211 14067 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 03:43:08.967779 14038 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 03:43:08.967984 14038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.054s user 0.004s sys 0.003s
I20250625 03:43:08.968175 14038 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 03:43:08.973778 14038 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 03:43:08.974110 14038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.006s sys 0.000s
I20250625 03:43:08.975414 14067 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Bootstrap starting.
W20250625 03:43:09.143824 13370 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): Couldn't send request to peer 975eeb611762460bb2f09f5b4b71e3d4. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250625 03:43:09.192723 14067 log.cc:826] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Log is configured to *not* fsync() on all Append() calls
W20250625 03:43:09.672425 13370 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): Couldn't send request to peer 975eeb611762460bb2f09f5b4b71e3d4. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250625 03:43:10.220988 13370 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): Couldn't send request to peer 975eeb611762460bb2f09f5b4b71e3d4. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20250625 03:43:10.253389 14180 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.130:34057 every 8 connection(s)
I20250625 03:43:10.253837 14038 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.130:34057
I20250625 03:43:10.256628 14038 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data/info.pb
I20250625 03:43:10.266993 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 14038
I20250625 03:43:10.267304 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 13088
I20250625 03:43:10.370725 13210 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 03:43:10.415230 14181 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:43:10.415810 14181 heartbeater.cc:461] Registering TS with master...
I20250625 03:43:10.417428 14181 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:10.425488 13020 ts_manager.cc:194] Re-registered known tserver with Master: 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:43:10.427899 13020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.130:43453
I20250625 03:43:10.708283 14067 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=204 ignored=0} inserts{seen=10150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250625 03:43:10.709612 14067 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Bootstrap complete.
I20250625 03:43:10.711550 14067 ts_tablet_manager.cc:1397] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Time spent bootstrapping tablet: real 1.737s user 1.178s sys 0.136s
I20250625 03:43:10.717725 14067 raft_consensus.cc:357] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:10.730050 14067 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975eeb611762460bb2f09f5b4b71e3d4, State: Initialized, Role: FOLLOWER
I20250625 03:43:10.731397 14067 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 204, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:10.754527 14067 ts_tablet_manager.cc:1428] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Time spent starting tablet: real 0.043s user 0.007s sys 0.017s
I20250625 03:43:10.755373 14181 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
I20250625 03:43:10.798528 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 13088
I20250625 03:43:10.828325 14134 raft_consensus.cc:1215] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.204->[1.205-1.404] Dedup: 1.205->[1.206-1.404]
I20250625 03:43:10.844677 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.129:45233
--local_ip_for_outbound_sockets=127.12.175.129
--tserver_master_addrs=127.12.175.190:43735
--webserver_port=34437
--webserver_interface=127.12.175.129
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250625 03:43:10.849275 13367 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 03:43:11.139839 14191 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 513d19d66cd54765940762233636fd31)
I20250625 03:43:11.140211 14191 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:11.147361 14191 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
W20250625 03:43:11.204363 14061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111)
W20250625 03:43:11.221589 14061 leader_election.cc:336] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111)
I20250625 03:43:11.246460 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 404 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:43:11.299158 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 404 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:43:11.301510 14063 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 975eeb611762460bb2f09f5b4b71e3d4; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:43:11.303124 14191 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 03:43:11.335038 13565 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 404 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
W20250625 03:43:11.405874 13367 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250625 03:43:11.963516 13367 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250625 03:43:12.411742 14196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:43:12.413040 14196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:43:12.413322 14196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:43:12.427747 14196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:43:12.428313 14196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.129
I20250625 03:43:12.453197 14196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.129:45233
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.175.129
--webserver_port=34437
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.129
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:43:12.456795 14196 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:43:12.463006 14196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:43:12.479257 13367 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250625 03:43:12.514238 14219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:12.516238 14218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:12.531764 14221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:43:12.545285 14196 server_base.cc:1048] running on GCE node
I20250625 03:43:12.548273 14196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:43:12.569203 14196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:43:12.570919 14196 hybrid_clock.cc:648] HybridClock initialized: now 1750822992570751 us; error 158 us; skew 500 ppm
I20250625 03:43:12.571552 14196 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:43:12.585445 14196 webserver.cc:469] Webserver started at http://127.12.175.129:34437/ using document root <none> and password file <none>
I20250625 03:43:12.592824 14196 fs_manager.cc:362] Metadata directory not provided
I20250625 03:43:12.593286 14196 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250625 03:43:12.612977 13367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20250625 03:43:12.621414 14196 fs_manager.cc:714] Time spent opening directory manager: real 0.023s user 0.009s sys 0.012s
I20250625 03:43:12.626562 14227 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:43:12.629521 14196 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250625 03:43:12.629920 14196 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal
uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f"
format_stamp: "Formatted at 2025-06-25 03:42:58 on dist-test-slave-ttt3"
I20250625 03:43:12.631191 14196 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:43:12.749205 14196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:43:12.750947 14196 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:43:12.751467 14196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:43:12.759493 14196 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:43:12.772441 14236 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 03:43:12.800341 14196 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 03:43:12.800515 14196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.031s user 0.003s sys 0.000s
I20250625 03:43:12.800689 14196 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 03:43:12.806749 14196 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 03:43:12.807045 14196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.004s sys 0.002s
I20250625 03:43:12.808393 14236 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Bootstrap starting.
W20250625 03:43:13.057173 13367 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250625 03:43:13.211107 14236 log.cc:826] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Log is configured to *not* fsync() on all Append() calls
W20250625 03:43:13.589699 13367 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20250625 03:43:13.837438 14347 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.129:45233 every 8 connection(s)
I20250625 03:43:13.837878 14196 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.129:45233
I20250625 03:43:13.840480 14196 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data/info.pb
I20250625 03:43:13.847927 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 14196
I20250625 03:43:14.041540 14348 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:43:14.042260 14348 heartbeater.cc:461] Registering TS with master...
I20250625 03:43:14.044288 14348 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:14.049355 13020 ts_manager.cc:194] Re-registered known tserver with Master: 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233)
I20250625 03:43:14.051561 13020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.129:52901
W20250625 03:43:14.283375 13367 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
W20250625 03:43:15.049666 13367 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250625 03:43:15.056715 14348 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
W20250625 03:43:15.646847 13367 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250625 03:43:16.250438 13367 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
I20250625 03:43:16.506783 14236 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Bootstrap replayed 1/1 log segments. Stats: ops{read=379 overwritten=0 applied=377 ignored=0} inserts{seen=18800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250625 03:43:16.508213 14236 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Bootstrap complete.
I20250625 03:43:16.513924 14236 ts_tablet_manager.cc:1397] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Time spent bootstrapping tablet: real 3.706s user 1.844s sys 0.164s
I20250625 03:43:16.522514 14236 raft_consensus.cc:357] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 1 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:16.539866 14236 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb1b4dfaea74515a9e5aab1fbc0870f, State: Initialized, Role: FOLLOWER
I20250625 03:43:16.541180 14236 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 377, Last appended: 1.379, Last appended by leader: 379, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:16.548735 14236 ts_tablet_manager.cc:1428] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f: Time spent starting tablet: real 0.034s user 0.018s sys 0.007s
I20250625 03:43:16.582674 13678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:16.587105 13414 tablet_service.cc:1430] Tablet server has 1 leaders and 3 scanners
I20250625 03:43:16.661092 14114 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:16.734097 13810 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:16.738554 14302 raft_consensus.cc:1215] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.378->[1.379-1.726] Dedup: 1.379->[1.380-1.726]
I20250625 03:43:16.822788 13546 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:17.318506 14399 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 513d19d66cd54765940762233636fd31)
I20250625 03:43:17.318858 14399 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:17.325367 14282 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:17.375326 14399 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:17.665068 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 2 candidate_status { last_received { term: 1 index: 726 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:43:17.666764 13698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 2 candidate_status { last_received { term: 1 index: 726 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:43:17.669044 14134 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 2 candidate_status { last_received { term: 1 index: 726 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:43:17.669167 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 2 candidate_status { last_received { term: 1 index: 726 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:43:17.670143 14229 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9cb1b4dfaea74515a9e5aab1fbc0870f; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:43:17.674990 14399 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 03:43:20.164799 13886 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 73967.4us
I20250625 03:43:20.884023 13020 ts_manager.cc:284] Unset tserver state for 9cb1b4dfaea74515a9e5aab1fbc0870f from MAINTENANCE_MODE
I20250625 03:43:20.948802 13023 ts_manager.cc:284] Unset tserver state for 975eeb611762460bb2f09f5b4b71e3d4 from MAINTENANCE_MODE
I20250625 03:43:20.965770 13743 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:20.971722 13611 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:21.262794 14181 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:21.341184 14348 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:21.453677 13875 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:21.485796 13479 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
W20250625 03:43:21.812566 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.068s user 0.000s sys 0.000s
W20250625 03:43:21.998770 13886 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 46033.7us
I20250625 03:43:23.213238 12990 maintenance_mode-itest.cc:745] Restarting batch of 2 tservers: 513d19d66cd54765940762233636fd31,d275425c3ef443f88abb12d24f4c2f2f
W20250625 03:43:23.310832 13886 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 46480.7us
W20250625 03:43:23.669535 13367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.129:45233, user_credentials={real_user=slave}} blocked reactor thread for 43560.7us
I20250625 03:43:24.517717 13023 ts_manager.cc:295] Set tserver state for 513d19d66cd54765940762233636fd31 to MAINTENANCE_MODE
I20250625 03:43:24.758615 13026 ts_manager.cc:295] Set tserver state for d275425c3ef443f88abb12d24f4c2f2f to MAINTENANCE_MODE
I20250625 03:43:27.084677 13414 tablet_service.cc:1423] Tablet server 513d19d66cd54765940762233636fd31 set to quiescing
I20250625 03:43:27.084992 13414 tablet_service.cc:1430] Tablet server has 1 leaders and 3 scanners
I20250625 03:43:27.086162 14540 raft_consensus.cc:991] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: : Instructing follower d275425c3ef443f88abb12d24f4c2f2f to start an election
I20250625 03:43:27.087781 14540 raft_consensus.cc:1079] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 1 LEADER]: Signalling peer d275425c3ef443f88abb12d24f4c2f2f to start an election
I20250625 03:43:27.089308 13564 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908"
dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f"
from {username='slave'} at 127.12.175.131:41425
I20250625 03:43:27.089679 13564 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250625 03:43:27.089843 13564 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 1 FOLLOWER]: Advancing to term 2
I20250625 03:43:27.093153 13564 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:27.101024 13564 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 2 election: Requested vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:43:27.121210 13566 raft_consensus.cc:1238] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 2 FOLLOWER]: Rejecting Update request from peer 513d19d66cd54765940762233636fd31 for earlier term 1. Current term is 2. Ops: [1.1198-1.1198]
I20250625 03:43:27.139200 14540 raft_consensus.cc:991] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: : Instructing follower 975eeb611762460bb2f09f5b4b71e3d4 to start an election
I20250625 03:43:27.139461 14540 raft_consensus.cc:1079] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 1 LEADER]: Signalling peer 975eeb611762460bb2f09f5b4b71e3d4 to start an election
I20250625 03:43:27.142231 14133 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908"
dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4"
from {username='slave'} at 127.12.175.131:45923
I20250625 03:43:27.142733 14133 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250625 03:43:27.143028 14133 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 1 FOLLOWER]: Advancing to term 2
I20250625 03:43:27.149703 14537 raft_consensus.cc:991] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: : Instructing follower 9cb1b4dfaea74515a9e5aab1fbc0870f to start an election
I20250625 03:43:27.150194 14540 raft_consensus.cc:1079] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 1 LEADER]: Signalling peer 9cb1b4dfaea74515a9e5aab1fbc0870f to start an election
I20250625 03:43:27.165113 14493 raft_consensus.cc:991] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: : Instructing follower 9cb1b4dfaea74515a9e5aab1fbc0870f to start an election
I20250625 03:43:27.165328 14493 raft_consensus.cc:1079] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 1 LEADER]: Signalling peer 9cb1b4dfaea74515a9e5aab1fbc0870f to start an election
I20250625 03:43:27.161551 14300 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908"
dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f"
from {username='slave'} at 127.12.175.131:49621
I20250625 03:43:27.166577 14300 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250625 03:43:27.166895 14301 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908"
dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f"
from {username='slave'} at 127.12.175.131:49621
I20250625 03:43:27.167338 14300 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 1 FOLLOWER]: Advancing to term 2
I20250625 03:43:27.173200 14536 consensus_queue.cc:1046] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 }, Status: INVALID_TERM, Last received: 1.1197, Next index: 1198, Last known committed idx: 1197, Time since last communication: 0.000s
I20250625 03:43:27.173609 14536 raft_consensus.cc:3053] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 1 LEADER]: Stepping down as leader of term 1
I20250625 03:43:27.173713 14536 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 513d19d66cd54765940762233636fd31, State: Running, Role: LEADER
I20250625 03:43:27.174053 14536 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1199, Committed index: 1199, Last appended: 1.1201, Last appended by leader: 1201, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:27.174721 14536 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 1 FOLLOWER]: Advancing to term 2
W20250625 03:43:27.178828 13394 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47074: Illegal state: replica 513d19d66cd54765940762233636fd31 is not leader of this config: current role FOLLOWER
I20250625 03:43:27.182188 14300 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:27.182843 14301 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250625 03:43:27.183004 14301 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 2 FOLLOWER]: Advancing to term 3
I20250625 03:43:27.186023 14301 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:27.186471 14302 raft_consensus.cc:1238] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Rejecting Update request from peer 513d19d66cd54765940762233636fd31 for earlier term 1. Current term is 3. Ops: [1.1201-1.1201]
I20250625 03:43:27.188508 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1197 } } ignore_live_leader: true dest_uuid: "506230f3ca974bf796583a6be750e38f"
I20250625 03:43:27.188777 13697 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 1 FOLLOWER]: Advancing to term 2
I20250625 03:43:27.190507 13698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: true dest_uuid: "506230f3ca974bf796583a6be750e38f"
I20250625 03:43:27.190738 14301 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 3 election: Requested vote from peers 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:27.191520 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: true dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f"
I20250625 03:43:27.191851 13566 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 2 FOLLOWER]: Advancing to term 3
I20250625 03:43:27.191937 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: true dest_uuid: "513d19d66cd54765940762233636fd31"
I20250625 03:43:27.192173 13434 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 2 FOLLOWER]: Advancing to term 3
I20250625 03:43:27.194950 13566 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1b4dfaea74515a9e5aab1fbc0870f in term 3.
I20250625 03:43:27.195493 13434 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cb1b4dfaea74515a9e5aab1fbc0870f for term 3 because replica has last-logged OpId of term: 1 index: 1201, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1200.
I20250625 03:43:27.196331 13697 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d275425c3ef443f88abb12d24f4c2f2f for term 2 because replica has last-logged OpId of term: 1 index: 1200, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1197.
I20250625 03:43:27.197266 14133 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:27.198352 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: true dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4"
I20250625 03:43:27.199141 14233 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4
I20250625 03:43:27.201154 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1197 } } ignore_live_leader: true dest_uuid: "513d19d66cd54765940762233636fd31"
I20250625 03:43:27.201467 13434 raft_consensus.cc:2366] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d275425c3ef443f88abb12d24f4c2f2f for earlier term 2. Current term is 3.
I20250625 03:43:27.202155 13500 leader_election.cc:400] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 2 election: Vote denied by peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d275425c3ef443f88abb12d24f4c2f2f for earlier term 2. Current term is 3.
I20250625 03:43:27.202265 13500 leader_election.cc:403] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250625 03:43:27.204931 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1197 } } ignore_live_leader: true dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4"
I20250625 03:43:27.206097 14301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1197 } } ignore_live_leader: true dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f"
I20250625 03:43:27.206661 14301 raft_consensus.cc:2366] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d275425c3ef443f88abb12d24f4c2f2f for earlier term 2. Current term is 3.
I20250625 03:43:27.207368 13499 leader_election.cc:400] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 2 election: Vote denied by peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d275425c3ef443f88abb12d24f4c2f2f for earlier term 2. Current term is 3.
I20250625 03:43:27.207899 14134 raft_consensus.cc:1238] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 2 FOLLOWER]: Rejecting Update request from peer 513d19d66cd54765940762233636fd31 for earlier term 1. Current term is 2. Ops: [1.1200-1.1200]
I20250625 03:43:27.215979 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1199 } } ignore_live_leader: true dest_uuid: "506230f3ca974bf796583a6be750e38f"
I20250625 03:43:27.216517 13697 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for term 2 because replica has last-logged OpId of term: 1 index: 1200, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1199.
I20250625 03:43:27.218508 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1199 } } ignore_live_leader: true dest_uuid: "513d19d66cd54765940762233636fd31"
I20250625 03:43:27.218904 13434 raft_consensus.cc:2366] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for earlier term 2. Current term is 3.
I20250625 03:43:27.219573 14133 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 2 election: Requested vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:27.219595 14062 leader_election.cc:400] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 2 election: Vote denied by peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for earlier term 2. Current term is 3.
I20250625 03:43:27.219842 14062 leader_election.cc:403] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250625 03:43:27.222718 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: true dest_uuid: "513d19d66cd54765940762233636fd31"
I20250625 03:43:27.222286 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: true dest_uuid: "506230f3ca974bf796583a6be750e38f"
I20250625 03:43:27.225966 14567 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 2 FOLLOWER]: Advancing to term 3
I20250625 03:43:27.227306 13697 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1b4dfaea74515a9e5aab1fbc0870f in term 2.
I20250625 03:43:27.227648 13434 raft_consensus.cc:2366] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cb1b4dfaea74515a9e5aab1fbc0870f for earlier term 2. Current term is 3.
I20250625 03:43:27.228313 14229 leader_election.cc:400] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 2 election: Vote denied by peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cb1b4dfaea74515a9e5aab1fbc0870f for earlier term 2. Current term is 3.
I20250625 03:43:27.228488 14229 leader_election.cc:403] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250625 03:43:27.229252 14300 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 2 election: Requested vote from peers 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:27.230504 14565 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d275425c3ef443f88abb12d24f4c2f2f for earlier term 2. Current term is 3.
I20250625 03:43:27.231171 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1199 } } ignore_live_leader: true dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f"
I20250625 03:43:27.231436 13566 raft_consensus.cc:2366] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for earlier term 2. Current term is 3.
I20250625 03:43:27.232009 14564 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250625 03:43:27.233040 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: true dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f"
I20250625 03:43:27.233263 13566 raft_consensus.cc:2366] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cb1b4dfaea74515a9e5aab1fbc0870f for earlier term 2. Current term is 3.
I20250625 03:43:27.233845 14133 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: true dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4"
I20250625 03:43:27.232246 14564 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cb1b4dfaea74515a9e5aab1fbc0870f for earlier term 2. Current term is 3.
I20250625 03:43:27.234956 14064 leader_election.cc:400] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 2 election: Vote denied by peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for earlier term 2. Current term is 3.
I20250625 03:43:27.256158 14567 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for earlier term 2. Current term is 3.
I20250625 03:43:27.257393 14133 raft_consensus.cc:2366] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cb1b4dfaea74515a9e5aab1fbc0870f for earlier term 2. Current term is 3.
I20250625 03:43:27.278612 14233 leader_election.cc:400] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 2 election: Vote denied by peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cb1b4dfaea74515a9e5aab1fbc0870f for earlier term 2. Current term is 3.
I20250625 03:43:27.279065 14233 leader_election.cc:400] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 2 election: Vote denied by peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cb1b4dfaea74515a9e5aab1fbc0870f for earlier term 2. Current term is 3.
W20250625 03:43:27.281348 14260 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:45316: Illegal state: replica 9cb1b4dfaea74515a9e5aab1fbc0870f is not leader of this config: current role FOLLOWER
I20250625 03:43:27.282678 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1199 } } ignore_live_leader: true dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f"
I20250625 03:43:27.282971 14300 raft_consensus.cc:2366] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for earlier term 2. Current term is 3.
I20250625 03:43:27.283987 14061 leader_election.cc:400] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 2 election: Vote denied by peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233) with higher term. Message: Invalid argument: T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for earlier term 2. Current term is 3.
W20250625 03:43:27.296308 13658 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59414: Illegal state: replica 506230f3ca974bf796583a6be750e38f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.312426 14094 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38976: Illegal state: replica 975eeb611762460bb2f09f5b4b71e3d4 is not leader of this config: current role FOLLOWER
W20250625 03:43:27.353041 13526 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55064: Illegal state: replica d275425c3ef443f88abb12d24f4c2f2f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.367600 13394 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47074: Illegal state: replica 513d19d66cd54765940762233636fd31 is not leader of this config: current role FOLLOWER
W20250625 03:43:27.380766 14260 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:45316: Illegal state: replica 9cb1b4dfaea74515a9e5aab1fbc0870f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.394163 13658 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59414: Illegal state: replica 506230f3ca974bf796583a6be750e38f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.408812 14094 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38976: Illegal state: replica 975eeb611762460bb2f09f5b4b71e3d4 is not leader of this config: current role FOLLOWER
W20250625 03:43:27.422914 13526 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55064: Illegal state: replica d275425c3ef443f88abb12d24f4c2f2f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.439103 13388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47074: Illegal state: replica 513d19d66cd54765940762233636fd31 is not leader of this config: current role FOLLOWER
W20250625 03:43:27.457006 14260 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:45316: Illegal state: replica 9cb1b4dfaea74515a9e5aab1fbc0870f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.471194 13658 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59414: Illegal state: replica 506230f3ca974bf796583a6be750e38f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.486711 14094 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38976: Illegal state: replica 975eeb611762460bb2f09f5b4b71e3d4 is not leader of this config: current role FOLLOWER
W20250625 03:43:27.487442 14536 raft_consensus.cc:668] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: failed to trigger leader election: Illegal state: leader elections are disabled
W20250625 03:43:27.503825 13526 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55064: Illegal state: replica d275425c3ef443f88abb12d24f4c2f2f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.523924 13388 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47074: Illegal state: replica 513d19d66cd54765940762233636fd31 is not leader of this config: current role FOLLOWER
W20250625 03:43:27.547302 14260 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:45316: Illegal state: replica 9cb1b4dfaea74515a9e5aab1fbc0870f is not leader of this config: current role FOLLOWER
I20250625 03:43:27.547781 14565 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 03:43:27.548045 14565 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:27.549142 14565 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:43:27.549814 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1197 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:43:27.549821 14133 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1197 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:43:27.550163 14133 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d275425c3ef443f88abb12d24f4c2f2f for term 4 because replica has last-logged OpId of term: 1 index: 1199, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1197.
I20250625 03:43:27.550163 14300 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d275425c3ef443f88abb12d24f4c2f2f for term 4 because replica has last-logged OpId of term: 1 index: 1200, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1197.
I20250625 03:43:27.550552 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1197 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:43:27.550892 13697 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d275425c3ef443f88abb12d24f4c2f2f for term 4 because replica has last-logged OpId of term: 1 index: 1200, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1197.
I20250625 03:43:27.551167 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1197 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:43:27.551465 13434 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d275425c3ef443f88abb12d24f4c2f2f for term 4 because replica has last-logged OpId of term: 1 index: 1201, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1197.
I20250625 03:43:27.551956 13500 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:43:27.552386 14565 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250625 03:43:27.567911 13658 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59414: Illegal state: replica 506230f3ca974bf796583a6be750e38f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.588068 14094 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38976: Illegal state: replica 975eeb611762460bb2f09f5b4b71e3d4 is not leader of this config: current role FOLLOWER
I20250625 03:43:27.591825 14567 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 03:43:27.592068 14567 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:27.593093 14567 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:27.593528 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1199 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:43:27.593888 14300 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for term 4 because replica has last-logged OpId of term: 1 index: 1200, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1199.
I20250625 03:43:27.593959 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1199 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:43:27.594193 13566 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975eeb611762460bb2f09f5b4b71e3d4 in term 3.
I20250625 03:43:27.594489 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1199 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:43:27.594537 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1199 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:43:27.594806 13434 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for term 4 because replica has last-logged OpId of term: 1 index: 1201, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1199.
I20250625 03:43:27.594856 13697 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975eeb611762460bb2f09f5b4b71e3d4 for term 4 because replica has last-logged OpId of term: 1 index: 1200, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1199.
I20250625 03:43:27.595634 14061 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 975eeb611762460bb2f09f5b4b71e3d4, d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:43:27.596050 14567 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250625 03:43:27.609892 13526 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55064: Illegal state: replica d275425c3ef443f88abb12d24f4c2f2f is not leader of this config: current role FOLLOWER
I20250625 03:43:27.636584 14575 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 03:43:27.638064 14575 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
W20250625 03:43:27.638298 13392 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47074: Illegal state: replica 513d19d66cd54765940762233636fd31 is not leader of this config: current role FOLLOWER
I20250625 03:43:27.645164 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "506230f3ca974bf796583a6be750e38f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:43:27.645581 13434 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 506230f3ca974bf796583a6be750e38f for term 3 because replica has last-logged OpId of term: 1 index: 1201, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1200.
I20250625 03:43:27.652053 14575 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:27.654703 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "506230f3ca974bf796583a6be750e38f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:43:27.655172 14300 raft_consensus.cc:2391] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 506230f3ca974bf796583a6be750e38f in current term 3: Already voted for candidate 9cb1b4dfaea74515a9e5aab1fbc0870f in this term.
I20250625 03:43:27.659245 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "506230f3ca974bf796583a6be750e38f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:43:27.659566 13566 raft_consensus.cc:2391] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 506230f3ca974bf796583a6be750e38f in current term 3: Already voted for candidate 9cb1b4dfaea74515a9e5aab1fbc0870f in this term.
I20250625 03:43:27.660631 13634 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 506230f3ca974bf796583a6be750e38f; no voters: 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:43:27.661150 14575 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 2 FOLLOWER]: Advancing to term 3
I20250625 03:43:27.661686 14133 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "506230f3ca974bf796583a6be750e38f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:43:27.662014 14133 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506230f3ca974bf796583a6be750e38f in term 3.
W20250625 03:43:27.667277 14260 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:45316: Illegal state: replica 9cb1b4dfaea74515a9e5aab1fbc0870f is not leader of this config: current role FOLLOWER
I20250625 03:43:27.669193 14575 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250625 03:43:27.681891 13546 tablet_service.cc:1423] Tablet server d275425c3ef443f88abb12d24f4c2f2f set to quiescing
I20250625 03:43:27.682189 13546 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 03:43:27.692720 13658 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59414: Illegal state: replica 506230f3ca974bf796583a6be750e38f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.721130 14094 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38976: Illegal state: replica 975eeb611762460bb2f09f5b4b71e3d4 is not leader of this config: current role FOLLOWER
W20250625 03:43:27.748760 13526 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55064: Illegal state: replica d275425c3ef443f88abb12d24f4c2f2f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.780256 13392 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47074: Illegal state: replica 513d19d66cd54765940762233636fd31 is not leader of this config: current role FOLLOWER
W20250625 03:43:27.813256 14260 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:45316: Illegal state: replica 9cb1b4dfaea74515a9e5aab1fbc0870f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.846122 13658 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59414: Illegal state: replica 506230f3ca974bf796583a6be750e38f is not leader of this config: current role FOLLOWER
I20250625 03:43:27.851938 14587 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 03:43:27.852237 14587 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:27.853426 14587 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:27.853880 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:43:27.854115 14133 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:43:27.854188 13697 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cb1b4dfaea74515a9e5aab1fbc0870f in term 3.
I20250625 03:43:27.854382 14133 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cb1b4dfaea74515a9e5aab1fbc0870f in term 3.
I20250625 03:43:27.854369 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:43:27.854635 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:43:27.854861 13434 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9cb1b4dfaea74515a9e5aab1fbc0870f for term 4 because replica has last-logged OpId of term: 1 index: 1201, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1200.
I20250625 03:43:27.854890 13566 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cb1b4dfaea74515a9e5aab1fbc0870f in term 3.
I20250625 03:43:27.855940 14233 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f; no voters: 513d19d66cd54765940762233636fd31
I20250625 03:43:27.859303 14587 raft_consensus.cc:2802] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250625 03:43:27.859453 14587 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 03:43:27.859556 14587 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 3 FOLLOWER]: Advancing to term 4
I20250625 03:43:27.862756 14587 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:27.863948 14587 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 4 election: Requested vote from peers 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:27.864172 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f"
I20250625 03:43:27.864406 13697 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 3 FOLLOWER]: Advancing to term 4
I20250625 03:43:27.864609 14133 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4"
I20250625 03:43:27.864722 13434 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31"
I20250625 03:43:27.864863 14133 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 3 FOLLOWER]: Advancing to term 4
I20250625 03:43:27.864943 13434 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 3 FOLLOWER]: Advancing to term 4
I20250625 03:43:27.866888 13697 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1b4dfaea74515a9e5aab1fbc0870f in term 4.
I20250625 03:43:27.867259 13566 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1200 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f"
I20250625 03:43:27.867431 13566 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 3 FOLLOWER]: Advancing to term 4
I20250625 03:43:27.868022 14133 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1b4dfaea74515a9e5aab1fbc0870f in term 4.
I20250625 03:43:27.870556 13434 raft_consensus.cc:2408] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cb1b4dfaea74515a9e5aab1fbc0870f for term 4 because replica has last-logged OpId of term: 1 index: 1201, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1200.
I20250625 03:43:27.871268 13566 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cb1b4dfaea74515a9e5aab1fbc0870f in term 4.
I20250625 03:43:27.871935 14232 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 506230f3ca974bf796583a6be750e38f, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f; no voters: 513d19d66cd54765940762233636fd31
I20250625 03:43:27.872547 14587 raft_consensus.cc:2802] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 4 FOLLOWER]: Leader election won for term 4
W20250625 03:43:27.878932 14093 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38976: Illegal state: replica 975eeb611762460bb2f09f5b4b71e3d4 is not leader of this config: current role FOLLOWER
I20250625 03:43:27.886137 14587 raft_consensus.cc:695] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [term 4 LEADER]: Becoming Leader. State: Replica: 9cb1b4dfaea74515a9e5aab1fbc0870f, State: Running, Role: LEADER
I20250625 03:43:27.886919 14587 consensus_queue.cc:237] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1199, Committed index: 1199, Last appended: 1.1200, Last appended by leader: 1200, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:27.892524 13026 catalog_manager.cc:5582] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f reported cstate change: term changed from 1 to 4, leader changed from 513d19d66cd54765940762233636fd31 (127.12.175.131) to 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129). New cstate: current_term: 4 leader_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } health_report { overall_health: UNKNOWN } } }
W20250625 03:43:27.911895 13526 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:55064: Illegal state: replica d275425c3ef443f88abb12d24f4c2f2f is not leader of this config: current role FOLLOWER
W20250625 03:43:27.947232 13387 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47074: Illegal state: replica 513d19d66cd54765940762233636fd31 is not leader of this config: current role FOLLOWER
I20250625 03:43:27.975550 13697 raft_consensus.cc:1273] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 4 FOLLOWER]: Refusing update from remote peer 9cb1b4dfaea74515a9e5aab1fbc0870f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1200. Preceding OpId from leader: term: 4 index: 1201. (index mismatch)
I20250625 03:43:27.976631 14587 consensus_queue.cc:1035] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Connected to new peer: Peer: permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1201, Last known committed idx: 1199, Time since last communication: 0.000s
I20250625 03:43:27.978232 13566 raft_consensus.cc:1273] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Refusing update from remote peer 9cb1b4dfaea74515a9e5aab1fbc0870f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1197. Preceding OpId from leader: term: 4 index: 1201. (index mismatch)
I20250625 03:43:27.979064 14588 consensus_queue.cc:1035] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1201, Last known committed idx: 1197, Time since last communication: 0.000s
I20250625 03:43:28.000708 13434 raft_consensus.cc:1273] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Refusing update from remote peer 9cb1b4dfaea74515a9e5aab1fbc0870f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1201. Preceding OpId from leader: term: 4 index: 1201. (term mismatch)
I20250625 03:43:28.000933 13434 pending_rounds.cc:85] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Aborting all ops after (but not including) 1200
I20250625 03:43:28.001034 13434 pending_rounds.cc:107] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Aborting uncommitted WRITE_OP operation due to leader change: 1.1201
W20250625 03:43:28.001408 13434 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47074: Aborted: Op aborted by new leader
I20250625 03:43:28.003083 14587 consensus_queue.cc:1035] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Connected to new peer: Peer: permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1201, Last known committed idx: 1199, Time since last communication: 0.000s
I20250625 03:43:28.035050 14133 raft_consensus.cc:1273] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Refusing update from remote peer 9cb1b4dfaea74515a9e5aab1fbc0870f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1199. Preceding OpId from leader: term: 4 index: 1201. (index mismatch)
I20250625 03:43:28.036123 14596 consensus_queue.cc:1035] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1201, Last known committed idx: 1197, Time since last communication: 0.000s
I20250625 03:43:30.366308 13414 tablet_service.cc:1423] Tablet server 513d19d66cd54765940762233636fd31 set to quiescing
I20250625 03:43:30.366664 13414 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:33.276775 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 13351
I20250625 03:43:33.355199 13474 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 03:43:33.847906 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 13351
W20250625 03:43:33.940860 14229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.175.131:40037 (error 108)
I20250625 03:43:33.959738 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.131:40037
--local_ip_for_outbound_sockets=127.12.175.131
--tserver_master_addrs=127.12.175.190:43735
--webserver_port=38451
--webserver_interface=127.12.175.131
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250625 03:43:33.990954 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 50355.2us
W20250625 03:43:33.994107 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250625 03:43:34.564580 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250625 03:43:35.074124 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250625 03:43:35.372511 13933 meta_cache.cc:1510] marking tablet server 513d19d66cd54765940762233636fd31 (127.12.175.131:40037) as failed
W20250625 03:43:35.373128 13933 meta_cache.cc:302] tablet 9126b7f95c9d434cb308581a7ce5b908: replica 513d19d66cd54765940762233636fd31 (127.12.175.131:40037) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111)
W20250625 03:43:35.600967 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250625 03:43:35.619720 14657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:43:35.624612 14657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:43:35.624840 14657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:43:35.698282 14657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:43:35.698853 14657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.131
I20250625 03:43:35.719291 14657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.131:40037
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.175.131
--webserver_port=38451
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.131
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:43:35.722893 14657 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:43:35.726511 14657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:43:35.840072 14671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:35.871045 14670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:43:35.897164 14657 server_base.cc:1048] running on GCE node
W20250625 03:43:35.900449 14673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:43:35.904441 14657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:43:35.918902 14657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:43:35.935755 14657 hybrid_clock.cc:648] HybridClock initialized: now 1750823015935643 us; error 64 us; skew 500 ppm
I20250625 03:43:35.936414 14657 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:43:35.968276 14657 webserver.cc:469] Webserver started at http://127.12.175.131:38451/ using document root <none> and password file <none>
I20250625 03:43:35.970274 14657 fs_manager.cc:362] Metadata directory not provided
I20250625 03:43:35.970638 14657 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:43:36.006553 14657 fs_manager.cc:714] Time spent opening directory manager: real 0.032s user 0.003s sys 0.012s
I20250625 03:43:36.023990 14679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.004s user 0.001s sys 0.000s
I20250625 03:43:36.030948 14657 fs_manager.cc:730] Time spent opening block manager: real 0.020s user 0.009s sys 0.003s
I20250625 03:43:36.031884 14657 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal
uuid: "513d19d66cd54765940762233636fd31"
format_stamp: "Formatted at 2025-06-25 03:42:59 on dist-test-slave-ttt3"
I20250625 03:43:36.033354 14657 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250625 03:43:36.111611 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250625 03:43:36.131296 14657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:43:36.132993 14657 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:43:36.133497 14657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:43:36.145318 14657 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:43:36.230419 14687 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 03:43:36.306700 14657 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 03:43:36.306897 14657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.145s user 0.012s sys 0.026s
I20250625 03:43:36.307085 14657 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 03:43:36.313979 14657 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 03:43:36.314280 14657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.006s sys 0.000s
I20250625 03:43:36.315934 14687 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Bootstrap starting.
W20250625 03:43:36.645778 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250625 03:43:37.192399 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20250625 03:43:37.629396 14687 log.cc:826] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Log is configured to *not* fsync() on all Append() calls
W20250625 03:43:37.781466 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250625 03:43:38.256978 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20250625 03:43:38.386253 14801 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.131:40037 every 8 connection(s)
I20250625 03:43:38.386716 14657 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.131:40037
I20250625 03:43:38.389775 14657 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data/info.pb
I20250625 03:43:38.393491 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 14657
I20250625 03:43:38.393779 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 13483
I20250625 03:43:38.470803 13606 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 03:43:38.492882 14802 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:43:38.493474 14802 heartbeater.cc:461] Registering TS with master...
I20250625 03:43:38.495186 14802 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:38.502763 13020 ts_manager.cc:194] Re-registered known tserver with Master: 513d19d66cd54765940762233636fd31 (127.12.175.131:40037)
I20250625 03:43:38.505385 13020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.131:56491
W20250625 03:43:39.048017 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
I20250625 03:43:39.134372 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 13483
W20250625 03:43:39.255070 14233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.175.132:42473 (error 108) [suppressed 41 similar messages]
I20250625 03:43:39.264472 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.132:42473
--local_ip_for_outbound_sockets=127.12.175.132
--tserver_master_addrs=127.12.175.190:43735
--webserver_port=39777
--webserver_interface=127.12.175.132
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250625 03:43:39.310585 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.132:42473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 03:43:39.512465 14802 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
W20250625 03:43:39.540969 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20250625 03:43:39.763048 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.132:42473: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250625 03:43:39.978619 14819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:43:39.979312 14819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:43:39.979555 14819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:43:39.993196 14819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:43:39.993674 14819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.132
I20250625 03:43:40.012336 14819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.132:42473
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.12.175.132
--webserver_port=39777
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.132
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:43:40.015317 14819 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:43:40.018370 14819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:43:40.076081 14826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:43:40.105381 14819 server_base.cc:1048] running on GCE node
W20250625 03:43:40.105741 14829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:43:40.105971 14827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:43:40.109669 14819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:43:40.115965 14819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:43:40.123492 14819 hybrid_clock.cc:648] HybridClock initialized: now 1750823020123308 us; error 163 us; skew 500 ppm
I20250625 03:43:40.124145 14819 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250625 03:43:40.139314 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20250625 03:43:40.194938 14819 webserver.cc:469] Webserver started at http://127.12.175.132:39777/ using document root <none> and password file <none>
I20250625 03:43:40.197120 14819 fs_manager.cc:362] Metadata directory not provided
I20250625 03:43:40.197522 14819 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250625 03:43:40.221508 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.132:42473: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250625 03:43:40.235234 14819 fs_manager.cc:714] Time spent opening directory manager: real 0.027s user 0.007s sys 0.001s
I20250625 03:43:40.248726 14837 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:43:40.251093 14819 fs_manager.cc:730] Time spent opening block manager: real 0.013s user 0.001s sys 0.003s
I20250625 03:43:40.251492 14819 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal
uuid: "d275425c3ef443f88abb12d24f4c2f2f"
format_stamp: "Formatted at 2025-06-25 03:43:00 on dist-test-slave-ttt3"
I20250625 03:43:40.253263 14819 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:43:40.315204 14819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:43:40.316756 14819 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:43:40.317210 14819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:43:40.331264 14819 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:43:40.345142 14844 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 03:43:40.358421 14819 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 03:43:40.358567 14819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.000s sys 0.004s
I20250625 03:43:40.358716 14819 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 03:43:40.367507 14819 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 03:43:40.367794 14819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.005s sys 0.001s
I20250625 03:43:40.369086 14844 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Bootstrap starting.
W20250625 03:43:40.575965 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250625 03:43:40.778280 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.132:42473: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250625 03:43:41.096812 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250625 03:43:41.299827 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.132:42473: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250625 03:43:41.663935 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
I20250625 03:43:41.710642 14615 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 513d19d66cd54765940762233636fd31 is lagging by at least 2 ops behind the committed index
I20250625 03:43:41.769697 14952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.132:42473 every 8 connection(s)
I20250625 03:43:41.770179 14819 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.132:42473
I20250625 03:43:41.772799 14819 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data/info.pb
I20250625 03:43:41.781658 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 14819
I20250625 03:43:41.880831 14953 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:43:41.881675 14953 heartbeater.cc:461] Registering TS with master...
I20250625 03:43:41.889190 14953 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:43:41.893663 13020 ts_manager.cc:194] Re-registered known tserver with Master: d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:41.895936 13020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.132:49799
W20250625 03:43:42.119642 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
W20250625 03:43:42.317842 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250625 03:43:42.605227 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250625 03:43:42.649863 14687 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Bootstrap replaying log segment 1/1 (1022.9K/2.34M this segment, stats: ops{read=578 overwritten=0 applied=578 ignored=0} inserts{seen=28850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250625 03:43:42.903384 14953 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
I20250625 03:43:42.905262 14844 log.cc:826] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Log is configured to *not* fsync() on all Append() calls
W20250625 03:43:43.122834 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
W20250625 03:43:43.466758 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250625 03:43:43.774207 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250625 03:43:44.028210 14844 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Append to log took a long time: real 0.064s user 0.004s sys 0.000s
W20250625 03:43:44.632489 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250625 03:43:44.770813 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20250625 03:43:44.771351 14687 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20250625 03:43:44.852909 14970 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 513d19d66cd54765940762233636fd31 is lagging by at least 147 ops behind the committed index [suppressed 14 similar messages]
W20250625 03:43:45.245956 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
I20250625 03:43:45.268849 13678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:45.362601 14733 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:45.393932 14114 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250625 03:43:45.417764 13810 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:45.474109 14282 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 03:43:45.508400 14886 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 03:43:45.652523 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250625 03:43:46.311127 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
W20250625 03:43:46.857551 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
W20250625 03:43:47.100330 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
I20250625 03:43:47.679653 14687 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Bootstrap replaying log segment 1/1 (1.84M/2.34M this segment, stats: ops{read=1065 overwritten=0 applied=1065 ignored=0} inserts{seen=53200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:43:47.804072 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.106s user 0.000s sys 0.000s
W20250625 03:43:47.992113 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.004s sys 0.000s
I20250625 03:43:48.045432 14844 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Bootstrap replaying log segment 1/1 (702.6K/2.72M this segment, stats: ops{read=408 overwritten=0 applied=407 ignored=0} inserts{seen=20300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:43:48.049296 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20250625 03:43:48.146056 15015 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 513d19d66cd54765940762233636fd31 is lagging by at least 256 ops behind the committed index [suppressed 14 similar messages]
I20250625 03:43:48.526916 15012 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer d275425c3ef443f88abb12d24f4c2f2f is lagging by at least 3 ops behind the committed index
W20250625 03:43:48.721655 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250625 03:43:50.458189 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250625 03:43:50.459407 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
I20250625 03:43:51.420876 15033 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 513d19d66cd54765940762233636fd31 is lagging by at least 330 ops behind the committed index [suppressed 6 similar messages]
W20250625 03:43:51.593706 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
I20250625 03:43:51.658690 15036 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer d275425c3ef443f88abb12d24f4c2f2f is lagging by at least 85 ops behind the committed index [suppressed 10 similar messages]
W20250625 03:43:51.832031 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
I20250625 03:43:52.671343 14687 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Bootstrap replaying log segment 1/1 (2.20M/2.34M this segment, stats: ops{read=1274 overwritten=1 applied=1272 ignored=0} inserts{seen=63500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:43:52.774384 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
W20250625 03:43:52.812165 13885 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.175.129:45233, user_credentials={real_user=slave}} blocked reactor thread for 44480.2us
I20250625 03:43:53.047073 14844 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Bootstrap replaying log segment 1/1 (1.16M/2.72M this segment, stats: ops{read=688 overwritten=0 applied=682 ignored=0} inserts{seen=34050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:43:53.508994 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250625 03:43:53.702642 14687 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Append to log took a long time: real 0.077s user 0.002s sys 0.000s
W20250625 03:43:53.735797 14229 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
W20250625 03:43:53.774055 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.078s user 0.003s sys 0.000s
I20250625 03:43:53.948654 14687 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Bootstrap replayed 1/1 log segments. Stats: ops{read=1360 overwritten=1 applied=1359 ignored=0} inserts{seen=67850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 03:43:53.953109 14687 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Bootstrap complete.
I20250625 03:43:54.002900 14687 ts_tablet_manager.cc:1397] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Time spent bootstrapping tablet: real 17.688s user 7.404s sys 0.796s
I20250625 03:43:54.012476 14687 raft_consensus.cc:357] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:54.014106 14687 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513d19d66cd54765940762233636fd31, State: Initialized, Role: FOLLOWER
I20250625 03:43:54.015645 14687 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1359, Last appended: 4.1359, Last appended by leader: 1359, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:54.063606 14687 ts_tablet_manager.cc:1428] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Time spent starting tablet: real 0.060s user 0.031s sys 0.008s
W20250625 03:43:54.246878 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
I20250625 03:43:54.724841 15070 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer d275425c3ef443f88abb12d24f4c2f2f is lagging by at least 203 ops behind the committed index [suppressed 11 similar messages]
W20250625 03:43:54.872588 15103 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Append to log took a long time: real 0.088s user 0.003s sys 0.004s
I20250625 03:43:55.073699 15082 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:43:55.074105 15082 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:55.077582 15082 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
W20250625 03:43:55.101627 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.094s user 0.004s sys 0.000s
I20250625 03:43:55.129019 15070 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 513d19d66cd54765940762233636fd31 is lagging by at least 47 ops behind the committed index [suppressed 15 similar messages]
I20250625 03:43:55.148278 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 1784 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:43:55.157532 13678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:55.180763 14114 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
W20250625 03:43:55.193630 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 43887us
I20250625 03:43:55.199345 13696 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 1784 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:43:55.203948 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 1784 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
W20250625 03:43:55.211201 14684 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:43:55.211582 14684 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:43:55.212194 15082 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:43:55.225270 14134 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 1784 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:43:55.229032 14733 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:55.236332 14282 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 03:43:55.244189 13810 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:43:55.370069 14886 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 03:43:55.523742 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250625 03:43:55.537750 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.082s user 0.004s sys 0.000s
I20250625 03:43:56.642311 15082 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:43:56.642585 15082 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:43:56.644227 15082 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:43:56.645278 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 2209 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:43:56.646152 13698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 2209 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:43:56.647481 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 2209 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:43:56.649078 14682 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:43:56.649601 15082 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:43:56.651684 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 2209 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
W20250625 03:43:56.653750 14684 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:43:56.661298 14753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.12.175.129:33631 (request call id 82) took 1370 ms. Trace:
I20250625 03:43:56.665320 14753 rpcz_store.cc:276] 0625 03:43:55.290825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:43:55.290896 (+ 71us) service_pool.cc:225] Handling call
0625 03:43:55.304051 (+ 13155us) raft_consensus.cc:1469] Updating replica for 425 ops
0625 03:43:55.601913 (+297862us) raft_consensus.cc:1532] Early marking committed up to index 0
0625 03:43:55.601921 (+ 8us) raft_consensus.cc:1537] Triggering prepare for 425 ops
0625 03:43:55.602168 (+ 247us) write_op.cc:270] Start()
0625 03:43:55.602227 (+ 59us) write_op.cc:276] Timestamp: P: 1750823021681549 usec, L: 0
0625 03:43:55.602569 (+ 342us) write_op.cc:270] Start()
0625 03:43:55.602597 (+ 28us) write_op.cc:276] Timestamp: P: 1750823021683180 usec, L: 0
0625 03:43:55.602819 (+ 222us) write_op.cc:270] Start()
0625 03:43:55.602863 (+ 44us) write_op.cc:276] Timestamp: P: 1750823021722292 usec, L: 0
0625 03:43:55.603034 (+ 171us) write_op.cc:270] Start()
0625 03:43:55.603079 (+ 45us) write_op.cc:276] Timestamp: P: 1750823021736548 usec, L: 0
0625 03:43:55.603265 (+ 186us) write_op.cc:270] Start()
0625 03:43:55.603288 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021742111 usec, L: 0
0625 03:43:55.603486 (+ 198us) write_op.cc:270] Start()
0625 03:43:55.603511 (+ 25us) write_op.cc:276] Timestamp: P: 1750823021754228 usec, L: 0
0625 03:43:55.603659 (+ 148us) write_op.cc:270] Start()
0625 03:43:55.603784 (+ 125us) write_op.cc:276] Timestamp: P: 1750823021789017 usec, L: 0
0625 03:43:55.604068 (+ 284us) write_op.cc:270] Start()
0625 03:43:55.604092 (+ 24us) write_op.cc:276] Timestamp: P: 1750823021833347 usec, L: 0
0625 03:43:55.604272 (+ 180us) write_op.cc:270] Start()
0625 03:43:55.604296 (+ 24us) write_op.cc:276] Timestamp: P: 1750823021856737 usec, L: 0
0625 03:43:55.604480 (+ 184us) write_op.cc:270] Start()
0625 03:43:55.604502 (+ 22us) write_op.cc:276] Timestamp: P: 1750823021864398 usec, L: 0
0625 03:43:55.604647 (+ 145us) write_op.cc:270] Start()
0625 03:43:55.604667 (+ 20us) write_op.cc:276] Timestamp: P: 1750823021878715 usec, L: 0
0625 03:43:55.604859 (+ 192us) write_op.cc:270] Start()
0625 03:43:55.604884 (+ 25us) write_op.cc:276] Timestamp: P: 1750823021907195 usec, L: 0
0625 03:43:55.605154 (+ 270us) write_op.cc:270] Start()
0625 03:43:55.605186 (+ 32us) write_op.cc:276] Timestamp: P: 1750823021930487 usec, L: 0
0625 03:43:55.605345 (+ 159us) write_op.cc:270] Start()
0625 03:43:55.605366 (+ 21us) write_op.cc:276] Timestamp: P: 1750823021960234 usec, L: 0
0625 03:43:55.605531 (+ 165us) write_op.cc:270] Start()
0625 03:43:55.605552 (+ 21us) write_op.cc:276] Timestamp: P: 1750823021971958 usec, L: 0
0625 03:43:55.638025 (+ 32473us) write_op.cc:270] Start()
0625 03:43:55.638068 (+ 43us) write_op.cc:276] Timestamp: P: 1750823021981178 usec, L: 0
0625 03:43:55.638397 (+ 329us) write_op.cc:270] Start()
0625 03:43:55.638424 (+ 27us) write_op.cc:276] Timestamp: P: 1750823021998368 usec, L: 0
0625 03:43:55.638644 (+ 220us) write_op.cc:270] Start()
0625 03:43:55.638666 (+ 22us) write_op.cc:276] Timestamp: P: 1750823022021477 usec, L: 0
0625 03:43:55.638809 (+ 143us) write_op.cc:270] Start()
0625 03:43:55.638830 (+ 21us) write_op.cc:276] Timestamp: P: 1750823022054753 usec, L: 0
0625 03:43:55.638997 (+ 167us) write_op.cc:270] Start()
0625 03:43:55.639055 (+ 58us) write_op.cc:276] Timestamp: P: 1750823022056381 usec, L: 0
0625 03:43:55.639263 (+ 208us) write_op.cc:270] Start()
0625 03:43:55.639286 (+ 23us) write_op.cc:276] Timestamp: P: 1750823022069291 usec, L: 0
0625 03:43:55.639430 (+ 144us) write_op.cc:270] Start()
0625 03:43:55.639451 (+ 21us) write_op.cc:276] Timestamp: P: 1750823022120214 usec, L: 0
0625 03:43:55.639625 (+ 174us) write_op.cc:270] Start()
0625 03:43:55.639646 (+ 21us) write_op.cc:276] Timestamp: P: 1750823022124887 usec, L: 0
0625 03:43:55.639871 (+ 225us) write_op.cc:270] Start()
0625 03:43:55.639898 (+ 27us) write_op.cc:276] Timestamp: P: 1750823022182362 usec, L: 0
0625 03:43:55.640116 (+ 218us) write_op.cc:270] Start()
0625 03:43:55.640141 (+ 25us) write_op.cc:276] Timestamp: P: 1750823022235008 usec, L: 0
0625 03:43:55.640379 (+ 238us) write_op.cc:270] Start()
0625 03:43:55.640402 (+ 23us) write_op.cc:276] Timestamp: P: 1750823022257488 usec, L: 0
0625 03:43:55.640543 (+ 141us) write_op.cc:270] Start()
0625 03:43:55.668047 (+ 27504us) write_op.cc:276] Timestamp: P: 1750823022280169 usec, L: 0
0625 03:43:55.668405 (+ 358us) write_op.cc:270] Start()
0625 03:43:55.668440 (+ 35us) write_op.cc:276] Timestamp: P: 1750823022295232 usec, L: 0
0625 03:43:55.668642 (+ 202us) write_op.cc:270] Start()
0625 03:43:55.668663 (+ 21us) write_op.cc:276] Timestamp: P: 1750823022332654 usec, L: 0
0625 03:43:55.668815 (+ 152us) write_op.cc:270] Start()
0625 03:43:55.668835 (+ 20us) write_op.cc:276] Timestamp: P: 1750823022353973 usec, L: 0
0625 03:43:55.668986 (+ 151us) write_op.cc:270] Start()
0625 03:43:55.669005 (+ 19us) write_op.cc:276] Timestamp: P: 1750823022374160 usec, L: 0
0625 03:43:55.669157 (+ 152us) write_op.cc:270] Start()
0625 03:43:55.669189 (+ 32us) write_op.cc:276] Timestamp: P: 1750823022386620 usec, L: 0
0625 03:43:55.669371 (+ 182us) write_op.cc:270] Start()
0625 03:43:55.669391 (+ 20us) write_op.cc:276] Timestamp: P: 1750823022424704 usec, L: 0
0625 03:43:55.669542 (+ 151us) write_op.cc:270] Start()
0625 03:43:55.669564 (+ 22us) write_op.cc:276] Timestamp: P: 1750823022455710 usec, L: 0
0625 03:43:55.669788 (+ 224us) write_op.cc:270] Start()
0625 03:43:55.669814 (+ 26us) write_op.cc:276] Timestamp: P: 1750823022456974 usec, L: 0
0625 03:43:55.669989 (+ 175us) write_op.cc:270] Start()
0625 03:43:55.670013 (+ 24us) write_op.cc:276] Timestamp: P: 1750823022498290 usec, L: 0
0625 03:43:55.670171 (+ 158us) write_op.cc:270] Start()
0625 03:43:55.670209 (+ 38us) write_op.cc:276] Timestamp: P: 1750823022533208 usec, L: 0
0625 03:43:55.670381 (+ 172us) write_op.cc:270] Start()
0625 03:43:55.670407 (+ 26us) write_op.cc:276] Timestamp: P: 1750823022534584 usec, L: 0
0625 03:43:55.670596 (+ 189us) write_op.cc:270] Start()
0625 03:43:55.670619 (+ 23us) write_op.cc:276] Timestamp: P: 1750823022558809 usec, L: 0
0625 03:43:55.670784 (+ 165us) write_op.cc:270] Start()
0625 03:43:55.670806 (+ 22us) write_op.cc:276] Timestamp: P: 1750823022572849 usec, L: 0
0625 03:43:55.671002 (+ 196us) write_op.cc:270] Start()
0625 03:43:55.671026 (+ 24us) write_op.cc:276] Timestamp: P: 1750823022595752 usec, L: 0
0625 03:43:55.671173 (+ 147us) write_op.cc:270] Start()
0625 03:43:55.671227 (+ 54us) write_op.cc:276] Timestamp: P: 1750823022609394 usec, L: 0
0625 03:43:55.671439 (+ 212us) write_op.cc:270] Start()
0625 03:43:55.671466 (+ 27us) write_op.cc:276] Timestamp: P: 1750823022614753 usec, L: 0
0625 03:43:55.697713 (+ 26247us) write_op.cc:270] Start()
0625 03:43:55.697757 (+ 44us) write_op.cc:276] Timestamp: P: 1750823022642612 usec, L: 0
0625 03:43:55.698042 (+ 285us) write_op.cc:270] Start()
0625 03:43:55.698073 (+ 31us) write_op.cc:276] Timestamp: P: 1750823022656979 usec, L: 0
0625 03:43:55.698441 (+ 368us) write_op.cc:270] Start()
0625 03:43:55.698475 (+ 34us) write_op.cc:276] Timestamp: P: 1750823022669706 usec, L: 0
0625 03:43:55.698674 (+ 199us) write_op.cc:270] Start()
0625 03:43:55.698702 (+ 28us) write_op.cc:276] Timestamp: P: 1750823022672325 usec, L: 0
0625 03:43:55.698895 (+ 193us) write_op.cc:270] Start()
0625 03:43:55.698920 (+ 25us) write_op.cc:276] Timestamp: P: 1750823022699654 usec, L: 0
0625 03:43:55.699113 (+ 193us) write_op.cc:270] Start()
0625 03:43:55.699138 (+ 25us) write_op.cc:276] Timestamp: P: 1750823022700867 usec, L: 0
0625 03:43:55.699332 (+ 194us) write_op.cc:270] Start()
0625 03:43:55.699357 (+ 25us) write_op.cc:276] Timestamp: P: 1750823022708965 usec, L: 0
0625 03:43:55.699532 (+ 175us) write_op.cc:270] Start()
0625 03:43:55.699554 (+ 22us) write_op.cc:276] Timestamp: P: 1750823022750248 usec, L: 0
0625 03:43:55.724867 (+ 25313us) write_op.cc:270] Start()
0625 03:43:55.724922 (+ 55us) write_op.cc:276] Timestamp: P: 1750823022751677 usec, L: 0
0625 03:43:55.729859 (+ 4937us) write_op.cc:270] Start()
0625 03:43:55.729922 (+ 63us) write_op.cc:276] Timestamp: P: 1750823022757546 usec, L: 0
0625 03:43:55.730239 (+ 317us) write_op.cc:270] Start()
0625 03:43:55.730270 (+ 31us) write_op.cc:276] Timestamp: P: 1750823022862985 usec, L: 0
0625 03:43:55.730470 (+ 200us) write_op.cc:270] Start()
0625 03:43:55.730498 (+ 28us) write_op.cc:276] Timestamp: P: 1750823022868282 usec, L: 0
0625 03:43:55.730712 (+ 214us) write_op.cc:270] Start()
0625 03:43:55.730736 (+ 24us) write_op.cc:276] Timestamp: P: 1750823022890790 usec, L: 0
0625 03:43:55.732309 (+ 1573us) write_op.cc:270] Start()
0625 03:43:55.732337 (+ 28us) write_op.cc:276] Timestamp: P: 1750823022901841 usec, L: 0
0625 03:43:55.732514 (+ 177us) write_op.cc:270] Start()
0625 03:43:55.732538 (+ 24us) write_op.cc:276] Timestamp: P: 1750823022905622 usec, L: 0
0625 03:43:55.732727 (+ 189us) write_op.cc:270] Start()
0625 03:43:55.749862 (+ 17135us) write_op.cc:276] Timestamp: P: 1750823022933541 usec, L: 0
0625 03:43:55.750259 (+ 397us) write_op.cc:270] Start()
0625 03:43:55.750317 (+ 58us) write_op.cc:276] Timestamp: P: 1750823022950652 usec, L: 0
0625 03:43:55.750598 (+ 281us) write_op.cc:270] Start()
0625 03:43:55.750626 (+ 28us) write_op.cc:276] Timestamp: P: 1750823022978478 usec, L: 0
0625 03:43:55.750868 (+ 242us) write_op.cc:270] Start()
0625 03:43:55.750896 (+ 28us) write_op.cc:276] Timestamp: P: 1750823022985722 usec, L: 0
0625 03:43:55.762468 (+ 11572us) write_op.cc:270] Start()
0625 03:43:55.762498 (+ 30us) write_op.cc:276] Timestamp: P: 1750823023007790 usec, L: 0
0625 03:43:55.762716 (+ 218us) write_op.cc:270] Start()
0625 03:43:55.762739 (+ 23us) write_op.cc:276] Timestamp: P: 1750823023014910 usec, L: 0
0625 03:43:55.762920 (+ 181us) write_op.cc:270] Start()
0625 03:43:55.762941 (+ 21us) write_op.cc:276] Timestamp: P: 1750823023028711 usec, L: 0
0625 03:43:55.763120 (+ 179us) write_op.cc:270] Start()
0625 03:43:55.763142 (+ 22us) write_op.cc:276] Timestamp: P: 1750823023034633 usec, L: 0
0625 03:43:55.763344 (+ 202us) write_op.cc:270] Start()
0625 03:43:55.763368 (+ 24us) write_op.cc:276] Timestamp: P: 1750823023064393 usec, L: 0
0625 03:43:55.763518 (+ 150us) write_op.cc:270] Start()
0625 03:43:55.763540 (+ 22us) write_op.cc:276] Timestamp: P: 1750823023081327 usec, L: 0
0625 03:43:55.763684 (+ 144us) write_op.cc:270] Start()
0625 03:43:55.763729 (+ 45us) write_op.cc:276] Timestamp: P: 1750823023083133 usec, L: 0
0625 03:43:55.763875 (+ 146us) write_op.cc:270] Start()
0625 03:43:55.763894 (+ 19us) write_op.cc:276] Timestamp: P: 1750823023124293 usec, L: 0
0625 03:43:55.764040 (+ 146us) write_op.cc:270] Start()
0625 03:43:55.764059 (+ 19us) write_op.cc:276] Timestamp: P: 1750823023148827 usec, L: 0
0625 03:43:55.764224 (+ 165us) write_op.cc:270] Start()
0625 03:43:55.764245 (+ 21us) write_op.cc:276] Timestamp: P: 1750823023193182 usec, L: 0
0625 03:43:55.764389 (+ 144us) write_op.cc:270] Start()
0625 03:43:55.764408 (+ 19us) write_op.cc:276] Timestamp: P: 1750823023220446 usec, L: 0
0625 03:43:55.764550 (+ 142us) write_op.cc:270] Start()
0625 03:43:55.764571 (+ 21us) write_op.cc:276] Timestamp: P: 1750823023248141 usec, L: 0
0625 03:43:55.764722 (+ 151us) write_op.cc:270] Start()
0625 03:43:55.764742 (+ 20us) write_op.cc:276] Timestamp: P: 1750823023272538 usec, L: 0
0625 03:43:55.796659 (+ 31917us) write_op.cc:270] Start()
0625 03:43:55.796708 (+ 49us) write_op.cc:276] Timestamp: P: 1750823023298761 usec, L: 0
0625 03:43:55.803596 (+ 6888us) write_op.cc:270] Start()
0625 03:43:55.803638 (+ 42us) write_op.cc:276] Timestamp: P: 1750823023328721 usec, L: 0
0625 03:43:55.803985 (+ 347us) write_op.cc:270] Start()
0625 03:43:55.804013 (+ 28us) write_op.cc:276] Timestamp: P: 1750823023330213 usec, L: 0
0625 03:43:55.804261 (+ 248us) write_op.cc:270] Start()
0625 03:43:55.804288 (+ 27us) write_op.cc:276] Timestamp: P: 1750823023366627 usec, L: 0
0625 03:43:55.804543 (+ 255us) write_op.cc:270] Start()
0625 03:43:55.804569 (+ 26us) write_op.cc:276] Timestamp: P: 1750823023368199 usec, L: 0
0625 03:43:55.804740 (+ 171us) write_op.cc:270] Start()
0625 03:43:55.804763 (+ 23us) write_op.cc:276] Timestamp: P: 1750823023390540 usec, L: 0
0625 03:43:55.804929 (+ 166us) write_op.cc:270] Start()
0625 03:43:55.804954 (+ 25us) write_op.cc:276] Timestamp: P: 1750823023412501 usec, L: 0
0625 03:43:55.805119 (+ 165us) write_op.cc:270] Start()
0625 03:43:55.805142 (+ 23us) write_op.cc:276] Timestamp: P: 1750823023420200 usec, L: 0
0625 03:43:55.805314 (+ 172us) write_op.cc:270] Start()
0625 03:43:55.805340 (+ 26us) write_op.cc:276] Timestamp: P: 1750823023427065 usec, L: 0
0625 03:43:55.805673 (+ 333us) write_op.cc:270] Start()
0625 03:43:55.805700 (+ 27us) write_op.cc:276] Timestamp: P: 1750823023460146 usec, L: 0
0625 03:43:55.805948 (+ 248us) write_op.cc:270] Start()
0625 03:43:55.805974 (+ 26us) write_op.cc:276] Timestamp: P: 1750823023474943 usec, L: 0
0625 03:43:55.806148 (+ 174us) write_op.cc:270] Start()
0625 03:43:55.806171 (+ 23us) write_op.cc:276] Timestamp: P: 1750823023501458 usec, L: 0
0625 03:43:55.806378 (+ 207us) write_op.cc:270] Start()
0625 03:43:55.806403 (+ 25us) write_op.cc:276] Timestamp: P: 1750823023530498 usec, L: 0
0625 03:43:55.806616 (+ 213us) write_op.cc:270] Start()
0625 03:43:55.806642 (+ 26us) write_op.cc:276] Timestamp: P: 1750823023534939 usec, L: 0
0625 03:43:55.806862 (+ 220us) write_op.cc:270] Start()
0625 03:43:55.806888 (+ 26us) write_op.cc:276] Timestamp: P: 1750823023579486 usec, L: 0
0625 03:43:55.807113 (+ 225us) write_op.cc:270] Start()
0625 03:43:55.807139 (+ 26us) write_op.cc:276] Timestamp: P: 1750823023584414 usec, L: 0
0625 03:43:55.807322 (+ 183us) write_op.cc:270] Start()
0625 03:43:55.807346 (+ 24us) write_op.cc:276] Timestamp: P: 1750823023585729 usec, L: 0
0625 03:43:55.807559 (+ 213us) write_op.cc:270] Start()
0625 03:43:55.807583 (+ 24us) write_op.cc:276] Timestamp: P: 1750823023631850 usec, L: 0
0625 03:43:55.842133 (+ 34550us) write_op.cc:270] Start()
0625 03:43:55.842176 (+ 43us) write_op.cc:276] Timestamp: P: 1750823023633440 usec, L: 0
0625 03:43:55.842492 (+ 316us) write_op.cc:270] Start()
0625 03:43:55.842518 (+ 26us) write_op.cc:276] Timestamp: P: 1750823023655083 usec, L: 0
0625 03:43:55.842731 (+ 213us) write_op.cc:270] Start()
0625 03:43:55.842756 (+ 25us) write_op.cc:276] Timestamp: P: 1750823023675063 usec, L: 0
0625 03:43:55.842941 (+ 185us) write_op.cc:270] Start()
0625 03:43:55.842963 (+ 22us) write_op.cc:276] Timestamp: P: 1750823023677449 usec, L: 0
0625 03:43:55.843108 (+ 145us) write_op.cc:270] Start()
0625 03:43:55.843126 (+ 18us) write_op.cc:276] Timestamp: P: 1750823023702362 usec, L: 0
0625 03:43:55.843330 (+ 204us) write_op.cc:270] Start()
0625 03:43:55.843355 (+ 25us) write_op.cc:276] Timestamp: P: 1750823023712401 usec, L: 0
0625 03:43:55.843512 (+ 157us) write_op.cc:270] Start()
0625 03:43:55.843531 (+ 19us) write_op.cc:276] Timestamp: P: 1750823023735000 usec, L: 0
0625 03:43:55.843720 (+ 189us) write_op.cc:270] Start()
0625 03:43:55.843742 (+ 22us) write_op.cc:276] Timestamp: P: 1750823023759407 usec, L: 0
0625 03:43:55.843903 (+ 161us) write_op.cc:270] Start()
0625 03:43:55.843922 (+ 19us) write_op.cc:276] Timestamp: P: 1750823023785277 usec, L: 0
0625 03:43:55.844079 (+ 157us) write_op.cc:270] Start()
0625 03:43:55.844097 (+ 18us) write_op.cc:276] Timestamp: P: 1750823023803248 usec, L: 0
0625 03:43:55.844278 (+ 181us) write_op.cc:270] Start()
0625 03:43:55.844300 (+ 22us) write_op.cc:276] Timestamp: P: 1750823023855045 usec, L: 0
0625 03:43:55.844511 (+ 211us) write_op.cc:270] Start()
0625 03:43:55.844533 (+ 22us) write_op.cc:276] Timestamp: P: 1750823023865661 usec, L: 0
0625 03:43:55.844681 (+ 148us) write_op.cc:270] Start()
0625 03:43:55.844699 (+ 18us) write_op.cc:276] Timestamp: P: 1750823023867000 usec, L: 0
0625 03:43:55.844872 (+ 173us) write_op.cc:270] Start()
0625 03:43:55.844893 (+ 21us) write_op.cc:276] Timestamp: P: 1750823023881045 usec, L: 0
0625 03:43:55.847500 (+ 2607us) write_op.cc:270] Start()
0625 03:43:55.847537 (+ 37us) write_op.cc:276] Timestamp: P: 1750823023908301 usec, L: 0
0625 03:43:55.847771 (+ 234us) write_op.cc:270] Start()
0625 03:43:55.847794 (+ 23us) write_op.cc:276] Timestamp: P: 1750823023927099 usec, L: 0
0625 03:43:55.848057 (+ 263us) write_op.cc:270] Start()
0625 03:43:55.848082 (+ 25us) write_op.cc:276] Timestamp: P: 1750823023928464 usec, L: 0
0625 03:43:55.848234 (+ 152us) write_op.cc:270] Start()
0625 03:43:55.848257 (+ 23us) write_op.cc:276] Timestamp: P: 1750823024000110 usec, L: 0
0625 03:43:55.848617 (+ 360us) write_op.cc:270] Start()
0625 03:43:55.848646 (+ 29us) write_op.cc:276] Timestamp: P: 1750823024014855 usec, L: 0
0625 03:43:55.848805 (+ 159us) write_op.cc:270] Start()
0625 03:43:55.848825 (+ 20us) write_op.cc:276] Timestamp: P: 1750823024021555 usec, L: 0
0625 03:43:55.849076 (+ 251us) write_op.cc:270] Start()
0625 03:43:55.849102 (+ 26us) write_op.cc:276] Timestamp: P: 1750823024050470 usec, L: 0
0625 03:43:55.853798 (+ 4696us) write_op.cc:270] Start()
0625 03:43:55.853838 (+ 40us) write_op.cc:276] Timestamp: P: 1750823024067387 usec, L: 0
0625 03:43:55.854076 (+ 238us) write_op.cc:270] Start()
0625 03:43:55.854102 (+ 26us) write_op.cc:276] Timestamp: P: 1750823024080679 usec, L: 0
0625 03:43:55.854300 (+ 198us) write_op.cc:270] Start()
0625 03:43:55.854322 (+ 22us) write_op.cc:276] Timestamp: P: 1750823024102724 usec, L: 0
0625 03:43:55.854514 (+ 192us) write_op.cc:270] Start()
0625 03:43:55.854539 (+ 25us) write_op.cc:276] Timestamp: P: 1750823024149252 usec, L: 0
0625 03:43:55.854738 (+ 199us) write_op.cc:270] Start()
0625 03:43:55.854797 (+ 59us) write_op.cc:276] Timestamp: P: 1750823024152575 usec, L: 0
0625 03:43:55.855018 (+ 221us) write_op.cc:270] Start()
0625 03:43:55.855039 (+ 21us) write_op.cc:276] Timestamp: P: 1750823024173964 usec, L: 0
0625 03:43:55.855192 (+ 153us) write_op.cc:270] Start()
0625 03:43:55.864551 (+ 9359us) write_op.cc:276] Timestamp: P: 1750823024199892 usec, L: 0
0625 03:43:55.864950 (+ 399us) write_op.cc:270] Start()
0625 03:43:55.864981 (+ 31us) write_op.cc:276] Timestamp: P: 1750823024207851 usec, L: 0
0625 03:43:55.865403 (+ 422us) write_op.cc:270] Start()
0625 03:43:55.865432 (+ 29us) write_op.cc:276] Timestamp: P: 1750823024235928 usec, L: 0
0625 03:43:55.865669 (+ 237us) write_op.cc:270] Start()
0625 03:43:55.865694 (+ 25us) write_op.cc:276] Timestamp: P: 1750823024237292 usec, L: 0
0625 03:43:55.865881 (+ 187us) write_op.cc:270] Start()
0625 03:43:55.865901 (+ 20us) write_op.cc:276] Timestamp: P: 1750823024266595 usec, L: 0
0625 03:43:55.866103 (+ 202us) write_op.cc:270] Start()
0625 03:43:55.866127 (+ 24us) write_op.cc:276] Timestamp: P: 1750823024290187 usec, L: 0
0625 03:43:55.866276 (+ 149us) write_op.cc:270] Start()
0625 03:43:55.866301 (+ 25us) write_op.cc:276] Timestamp: P: 1750823024311866 usec, L: 0
0625 03:43:55.866548 (+ 247us) write_op.cc:270] Start()
0625 03:43:55.866575 (+ 27us) write_op.cc:276] Timestamp: P: 1750823024337869 usec, L: 0
0625 03:43:55.866818 (+ 243us) write_op.cc:270] Start()
0625 03:43:55.866845 (+ 27us) write_op.cc:276] Timestamp: P: 1750823024350218 usec, L: 0
0625 03:43:55.867057 (+ 212us) write_op.cc:270] Start()
0625 03:43:55.867087 (+ 30us) write_op.cc:276] Timestamp: P: 1750823024377423 usec, L: 0
0625 03:43:55.867290 (+ 203us) write_op.cc:270] Start()
0625 03:43:55.867330 (+ 40us) write_op.cc:276] Timestamp: P: 1750823024412274 usec, L: 0
0625 03:43:55.867489 (+ 159us) write_op.cc:270] Start()
0625 03:43:55.867510 (+ 21us) write_op.cc:276] Timestamp: P: 1750823024444018 usec, L: 0
0625 03:43:55.867724 (+ 214us) write_op.cc:270] Start()
0625 03:43:55.867748 (+ 24us) write_op.cc:276] Timestamp: P: 1750823024450342 usec, L: 0
0625 03:43:55.867934 (+ 186us) write_op.cc:270] Start()
0625 03:43:55.867956 (+ 22us) write_op.cc:276] Timestamp: P: 1750823024500095 usec, L: 0
0625 03:43:55.868170 (+ 214us) write_op.cc:270] Start()
0625 03:43:55.868193 (+ 23us) write_op.cc:276] Timestamp: P: 1750823024501570 usec, L: 0
0625 03:43:55.868572 (+ 379us) write_op.cc:270] Start()
0625 03:43:55.868604 (+ 32us) write_op.cc:276] Timestamp: P: 1750823024527465 usec, L: 0
0625 03:43:55.868816 (+ 212us) write_op.cc:270] Start()
0625 03:43:55.868838 (+ 22us) write_op.cc:276] Timestamp: P: 1750823024556978 usec, L: 0
0625 03:43:55.869030 (+ 192us) write_op.cc:270] Start()
0625 03:43:55.869052 (+ 22us) write_op.cc:276] Timestamp: P: 1750823024587921 usec, L: 0
0625 03:43:55.869227 (+ 175us) write_op.cc:270] Start()
0625 03:43:55.869250 (+ 23us) write_op.cc:276] Timestamp: P: 1750823024611871 usec, L: 0
0625 03:43:55.869434 (+ 184us) write_op.cc:270] Start()
0625 03:43:55.869455 (+ 21us) write_op.cc:276] Timestamp: P: 1750823024617813 usec, L: 0
0625 03:43:55.869634 (+ 179us) write_op.cc:270] Start()
0625 03:43:55.869655 (+ 21us) write_op.cc:276] Timestamp: P: 1750823024651906 usec, L: 0
0625 03:43:55.869846 (+ 191us) write_op.cc:270] Start()
0625 03:43:55.869867 (+ 21us) write_op.cc:276] Timestamp: P: 1750823024658163 usec, L: 0
0625 03:43:55.870017 (+ 150us) write_op.cc:270] Start()
0625 03:43:55.870035 (+ 18us) write_op.cc:276] Timestamp: P: 1750823024668356 usec, L: 0
0625 03:43:55.870243 (+ 208us) write_op.cc:270] Start()
0625 03:43:55.870265 (+ 22us) write_op.cc:276] Timestamp: P: 1750823024757228 usec, L: 0
0625 03:43:55.870410 (+ 145us) write_op.cc:270] Start()
0625 03:43:55.870443 (+ 33us) write_op.cc:276] Timestamp: P: 1750823024759329 usec, L: 0
0625 03:43:55.870640 (+ 197us) write_op.cc:270] Start()
0625 03:43:55.870662 (+ 22us) write_op.cc:276] Timestamp: P: 1750823024777252 usec, L: 0
0625 03:43:55.870856 (+ 194us) write_op.cc:270] Start()
0625 03:43:55.870878 (+ 22us) write_op.cc:276] Timestamp: P: 1750823024824291 usec, L: 0
0625 03:43:55.871032 (+ 154us) write_op.cc:270] Start()
0625 03:43:55.871053 (+ 21us) write_op.cc:276] Timestamp: P: 1750823024854319 usec, L: 0
0625 03:43:55.871257 (+ 204us) write_op.cc:270] Start()
0625 03:43:55.871279 (+ 22us) write_op.cc:276] Timestamp: P: 1750823024863901 usec, L: 0
0625 03:43:55.871469 (+ 190us) write_op.cc:270] Start()
0625 03:43:55.871490 (+ 21us) write_op.cc:276] Timestamp: P: 1750823024900186 usec, L: 0
0625 03:43:55.871648 (+ 158us) write_op.cc:270] Start()
0625 03:43:55.871668 (+ 20us) write_op.cc:276] Timestamp: P: 1750823024936012 usec, L: 0
0625 03:43:55.920785 (+ 49117us) write_op.cc:270] Start()
0625 03:43:55.920822 (+ 37us) write_op.cc:276] Timestamp: P: 1750823024959812 usec, L: 0
0625 03:43:55.921112 (+ 290us) write_op.cc:270] Start()
0625 03:43:55.921164 (+ 52us) write_op.cc:276] Timestamp: P: 1750823024977473 usec, L: 0
0625 03:43:55.921551 (+ 387us) write_op.cc:270] Start()
0625 03:43:55.921578 (+ 27us) write_op.cc:276] Timestamp: P: 1750823025041437 usec, L: 0
0625 03:43:55.938206 (+ 16628us) write_op.cc:270] Start()
0625 03:43:55.938252 (+ 46us) write_op.cc:276] Timestamp: P: 1750823025066610 usec, L: 0
0625 03:43:55.938527 (+ 275us) write_op.cc:270] Start()
0625 03:43:55.938549 (+ 22us) write_op.cc:276] Timestamp: P: 1750823025104662 usec, L: 0
0625 03:43:55.938759 (+ 210us) write_op.cc:270] Start()
0625 03:43:55.938782 (+ 23us) write_op.cc:276] Timestamp: P: 1750823025126953 usec, L: 0
0625 03:43:55.938977 (+ 195us) write_op.cc:270] Start()
0625 03:43:55.938999 (+ 22us) write_op.cc:276] Timestamp: P: 1750823025161009 usec, L: 0
0625 03:43:55.939138 (+ 139us) write_op.cc:270] Start()
0625 03:43:55.939155 (+ 17us) write_op.cc:276] Timestamp: P: 1750823025186520 usec, L: 0
0625 03:43:55.939389 (+ 234us) write_op.cc:270] Start()
0625 03:43:55.939497 (+ 108us) write_op.cc:276] Timestamp: P: 1750823025187803 usec, L: 0
0625 03:43:55.939825 (+ 328us) write_op.cc:270] Start()
0625 03:43:55.939882 (+ 57us) write_op.cc:276] Timestamp: P: 1750823025211753 usec, L: 0
0625 03:43:55.940081 (+ 199us) write_op.cc:270] Start()
0625 03:43:55.940106 (+ 25us) write_op.cc:276] Timestamp: P: 1750823025248818 usec, L: 0
0625 03:43:55.940334 (+ 228us) write_op.cc:270] Start()
0625 03:43:55.940359 (+ 25us) write_op.cc:276] Timestamp: P: 1750823025269695 usec, L: 0
0625 03:43:55.940612 (+ 253us) write_op.cc:270] Start()
0625 03:43:55.940636 (+ 24us) write_op.cc:276] Timestamp: P: 1750823025394330 usec, L: 0
0625 03:43:55.940782 (+ 146us) write_op.cc:270] Start()
0625 03:43:55.940833 (+ 51us) write_op.cc:276] Timestamp: P: 1750823025400449 usec, L: 0
0625 03:43:55.941000 (+ 167us) write_op.cc:270] Start()
0625 03:43:55.941023 (+ 23us) write_op.cc:276] Timestamp: P: 1750823025437412 usec, L: 0
0625 03:43:55.965247 (+ 24224us) write_op.cc:270] Start()
0625 03:43:55.965290 (+ 43us) write_op.cc:276] Timestamp: P: 1750823025449311 usec, L: 0
0625 03:43:55.987955 (+ 22665us) write_op.cc:270] Start()
0625 03:43:55.988004 (+ 49us) write_op.cc:276] Timestamp: P: 1750823025460682 usec, L: 0
0625 03:43:55.988337 (+ 333us) write_op.cc:270] Start()
0625 03:43:55.988367 (+ 30us) write_op.cc:276] Timestamp: P: 1750823025512728 usec, L: 0
0625 03:43:55.988581 (+ 214us) write_op.cc:270] Start()
0625 03:43:55.988606 (+ 25us) write_op.cc:276] Timestamp: P: 1750823025515490 usec, L: 0
0625 03:43:55.988811 (+ 205us) write_op.cc:270] Start()
0625 03:43:55.988836 (+ 25us) write_op.cc:276] Timestamp: P: 1750823025540078 usec, L: 0
0625 03:43:55.989022 (+ 186us) write_op.cc:270] Start()
0625 03:43:55.989047 (+ 25us) write_op.cc:276] Timestamp: P: 1750823025593632 usec, L: 0
0625 03:43:55.989215 (+ 168us) write_op.cc:270] Start()
0625 03:43:55.989241 (+ 26us) write_op.cc:276] Timestamp: P: 1750823025602566 usec, L: 0
0625 03:43:55.989413 (+ 172us) write_op.cc:270] Start()
0625 03:43:55.989434 (+ 21us) write_op.cc:276] Timestamp: P: 1750823025621875 usec, L: 0
0625 03:43:55.989596 (+ 162us) write_op.cc:270] Start()
0625 03:43:55.989621 (+ 25us) write_op.cc:276] Timestamp: P: 1750823025668419 usec, L: 0
0625 03:43:55.989780 (+ 159us) write_op.cc:270] Start()
0625 03:43:55.989800 (+ 20us) write_op.cc:276] Timestamp: P: 1750823025677377 usec, L: 0
0625 03:43:55.989948 (+ 148us) write_op.cc:270] Start()
0625 03:43:55.989967 (+ 19us) write_op.cc:276] Timestamp: P: 1750823025739333 usec, L: 0
0625 03:43:55.990101 (+ 134us) write_op.cc:270] Start()
0625 03:43:55.990120 (+ 19us) write_op.cc:276] Timestamp: P: 1750823025809157 usec, L: 0
0625 03:43:55.990285 (+ 165us) write_op.cc:270] Start()
0625 03:43:55.990308 (+ 23us) write_op.cc:276] Timestamp: P: 1750823025818416 usec, L: 0
0625 03:43:55.990472 (+ 164us) write_op.cc:270] Start()
0625 03:43:55.990496 (+ 24us) write_op.cc:276] Timestamp: P: 1750823025908647 usec, L: 0
0625 03:43:55.990660 (+ 164us) write_op.cc:270] Start()
0625 03:43:55.990681 (+ 21us) write_op.cc:276] Timestamp: P: 1750823025912172 usec, L: 0
0625 03:43:55.990842 (+ 161us) write_op.cc:270] Start()
0625 03:43:55.990863 (+ 21us) write_op.cc:276] Timestamp: P: 1750823025927240 usec, L: 0
0625 03:43:55.991042 (+ 179us) write_op.cc:270] Start()
0625 03:43:55.991065 (+ 23us) write_op.cc:276] Timestamp: P: 1750823025951024 usec, L: 0
0625 03:43:55.991241 (+ 176us) write_op.cc:270] Start()
0625 03:43:55.991264 (+ 23us) write_op.cc:276] Timestamp: P: 1750823025967182 usec, L: 0
0625 03:43:55.991447 (+ 183us) write_op.cc:270] Start()
0625 03:43:55.991472 (+ 25us) write_op.cc:276] Timestamp: P: 1750823026005018 usec, L: 0
0625 03:43:55.991623 (+ 151us) write_op.cc:270] Start()
0625 03:43:55.991642 (+ 19us) write_op.cc:276] Timestamp: P: 1750823026035951 usec, L: 0
0625 03:43:56.006679 (+ 15037us) write_op.cc:270] Start()
0625 03:43:56.006727 (+ 48us) write_op.cc:276] Timestamp: P: 1750823026072920 usec, L: 0
0625 03:43:56.007000 (+ 273us) write_op.cc:270] Start()
0625 03:43:56.007024 (+ 24us) write_op.cc:276] Timestamp: P: 1750823026074276 usec, L: 0
0625 03:43:56.007192 (+ 168us) write_op.cc:270] Start()
0625 03:43:56.007220 (+ 28us) write_op.cc:276] Timestamp: P: 1750823026075492 usec, L: 0
0625 03:43:56.007405 (+ 185us) write_op.cc:270] Start()
0625 03:43:56.007430 (+ 25us) write_op.cc:276] Timestamp: P: 1750823026132716 usec, L: 0
0625 03:43:56.007640 (+ 210us) write_op.cc:270] Start()
0625 03:43:56.007668 (+ 28us) write_op.cc:276] Timestamp: P: 1750823026134018 usec, L: 0
0625 03:43:56.007886 (+ 218us) write_op.cc:270] Start()
0625 03:43:56.007911 (+ 25us) write_op.cc:276] Timestamp: P: 1750823026179512 usec, L: 0
0625 03:43:56.011160 (+ 3249us) write_op.cc:270] Start()
0625 03:43:56.011206 (+ 46us) write_op.cc:276] Timestamp: P: 1750823026205479 usec, L: 0
0625 03:43:56.011538 (+ 332us) write_op.cc:270] Start()
0625 03:43:56.011564 (+ 26us) write_op.cc:276] Timestamp: P: 1750823026216886 usec, L: 0
0625 03:43:56.011811 (+ 247us) write_op.cc:270] Start()
0625 03:43:56.011837 (+ 26us) write_op.cc:276] Timestamp: P: 1750823026255796 usec, L: 0
0625 03:43:56.012063 (+ 226us) write_op.cc:270] Start()
0625 03:43:56.012090 (+ 27us) write_op.cc:276] Timestamp: P: 1750823026280783 usec, L: 0
0625 03:43:56.012260 (+ 170us) write_op.cc:270] Start()
0625 03:43:56.012279 (+ 19us) write_op.cc:276] Timestamp: P: 17508230
W20250625 03:43:57.011901 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
W20250625 03:43:57.720758 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20250625 03:43:57.782352 15132 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer d275425c3ef443f88abb12d24f4c2f2f is lagging by at least 324 ops behind the committed index [suppressed 12 similar messages]
I20250625 03:43:58.056550 14844 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Bootstrap replaying log segment 1/1 (1.83M/2.72M this segment, stats: ops{read=1089 overwritten=0 applied=1087 ignored=0} inserts{seen=54300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:43:58.653535 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250625 03:43:59.092298 15103 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31: Append to log took a long time: real 0.069s user 0.003s sys 0.000s
W20250625 03:43:59.151504 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.060s user 0.004s sys 0.000s
W20250625 03:43:59.688323 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.065s user 0.001s sys 0.000s
W20250625 03:44:00.182965 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.127s user 0.000s sys 0.000s
W20250625 03:44:00.700320 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
I20250625 03:44:00.791057 15146 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer d275425c3ef443f88abb12d24f4c2f2f is lagging by at least 378 ops behind the committed index [suppressed 9 similar messages]
W20250625 03:44:01.648030 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 57562.1us
W20250625 03:44:01.778901 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 66479.3us
W20250625 03:44:01.942013 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.082s user 0.000s sys 0.003s
W20250625 03:44:02.627835 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20250625 03:44:03.059442 14844 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Bootstrap replaying log segment 1/1 (2.32M/2.72M this segment, stats: ops{read=1381 overwritten=0 applied=1381 ignored=0} inserts{seen=68950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:44:03.732549 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20250625 03:44:03.980399 15072 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer d275425c3ef443f88abb12d24f4c2f2f is lagging by at least 483 ops behind the committed index [suppressed 10 similar messages]
W20250625 03:44:04.678136 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250625 03:44:05.223436 14844 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Append to log took a long time: real 0.220s user 0.004s sys 0.000s
W20250625 03:44:05.364122 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.086s user 0.000s sys 0.000s
I20250625 03:44:05.952603 14733 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:44:05.974182 13810 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:44:05.985324 14114 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
W20250625 03:44:06.010629 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 51734.8us
W20250625 03:44:06.024713 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 44433.5us
I20250625 03:44:06.035591 13678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:44:06.091578 14886 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:44:06.233330 14282 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250625 03:44:06.990068 14233 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Couldn't send request to peer d275425c3ef443f88abb12d24f4c2f2f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
I20250625 03:44:07.037153 15219 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer d275425c3ef443f88abb12d24f4c2f2f is lagging by at least 545 ops behind the committed index [suppressed 8 similar messages]
I20250625 03:44:07.227234 14844 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1618 overwritten=0 applied=1617 ignored=0} inserts{seen=80750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250625 03:44:07.255754 14844 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Bootstrap complete.
I20250625 03:44:07.259541 14844 ts_tablet_manager.cc:1397] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Time spent bootstrapping tablet: real 26.891s user 10.166s sys 0.866s
I20250625 03:44:07.277693 14844 raft_consensus.cc:357] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:44:07.320931 14844 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: d275425c3ef443f88abb12d24f4c2f2f, State: Initialized, Role: FOLLOWER
I20250625 03:44:07.325618 14844 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1617, Last appended: 4.1618, Last appended by leader: 1618, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:44:07.337744 14844 ts_tablet_manager.cc:1428] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Time spent starting tablet: real 0.078s user 0.039s sys 0.015s
I20250625 03:44:07.650224 15227 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 03:44:07.651569 15227 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:44:07.716921 15227 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:44:07.795382 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 1618 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:44:07.800213 14134 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 1618 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:44:07.782117 14752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 1618 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:44:07.824442 14839 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:44:07.873767 13698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 1618 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:44:08.755175 15227 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:44:08.755548 14906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.12.175.129:45479 (request call id 108) took 1080 ms. Trace:
I20250625 03:44:09.069204 15227 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:44:09.070250 15227 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:44:09.101166 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2043 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:44:09.101802 13698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2043 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:44:09.103124 15227 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:44:09.104065 14134 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2043 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:44:09.105370 14841 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:44:09.105603 14753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2043 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:44:09.123975 15227 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:44:08.756103 14906 rpcz_store.cc:276] 0625 03:44:07.675582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:44:07.675680 (+ 98us) service_pool.cc:225] Handling call
0625 03:44:07.739809 (+ 64129us) raft_consensus.cc:1469] Updating replica for 425 ops
0625 03:44:07.785715 (+ 45906us) raft_consensus.cc:1532] Early marking committed up to index 1618
0625 03:44:07.810758 (+ 25043us) raft_consensus.cc:1537] Triggering prepare for 425 ops
0625 03:44:07.815619 (+ 4861us) write_op.cc:270] Start()
0625 03:44:07.815743 (+ 124us) write_op.cc:276] Timestamp: P: 1750823018449142 usec, L: 0
0625 03:44:07.818411 (+ 2668us) write_op.cc:270] Start()
0625 03:44:07.818476 (+ 65us) write_op.cc:276] Timestamp: P: 1750823018471667 usec, L: 0
0625 03:44:07.820120 (+ 1644us) write_op.cc:270] Start()
0625 03:44:07.820148 (+ 28us) write_op.cc:276] Timestamp: P: 1750823018519100 usec, L: 0
0625 03:44:07.821630 (+ 1482us) write_op.cc:270] Start()
0625 03:44:07.821661 (+ 31us) write_op.cc:276] Timestamp: P: 1750823018564701 usec, L: 0
0625 03:44:07.823360 (+ 1699us) write_op.cc:270] Start()
0625 03:44:07.823389 (+ 29us) write_op.cc:276] Timestamp: P: 1750823018573541 usec, L: 0
0625 03:44:07.824692 (+ 1303us) write_op.cc:270] Start()
0625 03:44:07.824731 (+ 39us) write_op.cc:276] Timestamp: P: 1750823018577938 usec, L: 0
0625 03:44:07.825938 (+ 1207us) write_op.cc:270] Start()
0625 03:44:07.825987 (+ 49us) write_op.cc:276] Timestamp: P: 1750823018597374 usec, L: 0
0625 03:44:07.827296 (+ 1309us) write_op.cc:270] Start()
0625 03:44:07.827322 (+ 26us) write_op.cc:276] Timestamp: P: 1750823018624645 usec, L: 0
0625 03:44:07.828732 (+ 1410us) write_op.cc:270] Start()
0625 03:44:07.828761 (+ 29us) write_op.cc:276] Timestamp: P: 1750823018628926 usec, L: 0
0625 03:44:07.830192 (+ 1431us) write_op.cc:270] Start()
0625 03:44:07.830230 (+ 38us) write_op.cc:276] Timestamp: P: 1750823018649065 usec, L: 0
0625 03:44:07.830449 (+ 219us) write_op.cc:270] Start()
0625 03:44:07.830472 (+ 23us) write_op.cc:276] Timestamp: P: 1750823018657706 usec, L: 0
0625 03:44:07.830631 (+ 159us) write_op.cc:270] Start()
0625 03:44:07.830654 (+ 23us) write_op.cc:276] Timestamp: P: 1750823018679386 usec, L: 0
0625 03:44:07.859654 (+ 29000us) write_op.cc:270] Start()
0625 03:44:07.859739 (+ 85us) write_op.cc:276] Timestamp: P: 1750823018697333 usec, L: 0
0625 03:44:07.859966 (+ 227us) write_op.cc:270] Start()
0625 03:44:07.860009 (+ 43us) write_op.cc:276] Timestamp: P: 1750823018708039 usec, L: 0
0625 03:44:07.860408 (+ 399us) write_op.cc:270] Start()
0625 03:44:07.860433 (+ 25us) write_op.cc:276] Timestamp: P: 1750823018732165 usec, L: 0
0625 03:44:07.860638 (+ 205us) write_op.cc:270] Start()
0625 03:44:07.860677 (+ 39us) write_op.cc:276] Timestamp: P: 1750823018749169 usec, L: 0
0625 03:44:07.898015 (+ 37338us) write_op.cc:270] Start()
0625 03:44:07.898187 (+ 172us) write_op.cc:276] Timestamp: P: 1750823018766475 usec, L: 0
0625 03:44:07.903320 (+ 5133us) write_op.cc:270] Start()
0625 03:44:07.903374 (+ 54us) write_op.cc:276] Timestamp: P: 1750823018771780 usec, L: 0
0625 03:44:07.903657 (+ 283us) write_op.cc:270] Start()
0625 03:44:07.903680 (+ 23us) write_op.cc:276] Timestamp: P: 1750823018788417 usec, L: 0
0625 03:44:07.903855 (+ 175us) write_op.cc:270] Start()
0625 03:44:07.904065 (+ 210us) write_op.cc:276] Timestamp: P: 1750823018820769 usec, L: 0
0625 03:44:07.904222 (+ 157us) write_op.cc:270] Start()
0625 03:44:07.904241 (+ 19us) write_op.cc:276] Timestamp: P: 1750823018826943 usec, L: 0
0625 03:44:07.904390 (+ 149us) write_op.cc:270] Start()
0625 03:44:07.904410 (+ 20us) write_op.cc:276] Timestamp: P: 1750823018865254 usec, L: 0
0625 03:44:07.904555 (+ 145us) write_op.cc:270] Start()
0625 03:44:07.904573 (+ 18us) write_op.cc:276] Timestamp: P: 1750823018866544 usec, L: 0
0625 03:44:07.904693 (+ 120us) write_op.cc:270] Start()
0625 03:44:07.904709 (+ 16us) write_op.cc:276] Timestamp: P: 1750823018872014 usec, L: 0
0625 03:44:07.904861 (+ 152us) write_op.cc:270] Start()
0625 03:44:07.904878 (+ 17us) write_op.cc:276] Timestamp: P: 1750823018926261 usec, L: 0
0625 03:44:07.905052 (+ 174us) write_op.cc:270] Start()
0625 03:44:07.905077 (+ 25us) write_op.cc:276] Timestamp: P: 1750823018931811 usec, L: 0
0625 03:44:07.905236 (+ 159us) write_op.cc:270] Start()
0625 03:44:07.905253 (+ 17us) write_op.cc:276] Timestamp: P: 1750823018964943 usec, L: 0
0625 03:44:07.905398 (+ 145us) write_op.cc:270] Start()
0625 03:44:07.905414 (+ 16us) write_op.cc:276] Timestamp: P: 1750823018999502 usec, L: 0
0625 03:44:07.905520 (+ 106us) write_op.cc:270] Start()
0625 03:44:07.905534 (+ 14us) write_op.cc:276] Timestamp: P: 1750823019027725 usec, L: 0
0625 03:44:07.905640 (+ 106us) write_op.cc:270] Start()
0625 03:44:07.905654 (+ 14us) write_op.cc:276] Timestamp: P: 1750823019047753 usec, L: 0
0625 03:44:07.905839 (+ 185us) write_op.cc:270] Start()
0625 03:44:07.905867 (+ 28us) write_op.cc:276] Timestamp: P: 1750823019086185 usec, L: 0
0625 03:44:07.906019 (+ 152us) write_op.cc:270] Start()
0625 03:44:07.906039 (+ 20us) write_op.cc:276] Timestamp: P: 1750823019087618 usec, L: 0
0625 03:44:07.906224 (+ 185us) write_op.cc:270] Start()
0625 03:44:07.906245 (+ 21us) write_op.cc:276] Timestamp: P: 1750823019137194 usec, L: 0
0625 03:44:07.906411 (+ 166us) write_op.cc:270] Start()
0625 03:44:07.906427 (+ 16us) write_op.cc:276] Timestamp: P: 1750823019172133 usec, L: 0
0625 03:44:07.906558 (+ 131us) write_op.cc:270] Start()
0625 03:44:07.906577 (+ 19us) write_op.cc:276] Timestamp: P: 1750823019196699 usec, L: 0
0625 03:44:07.906707 (+ 130us) write_op.cc:270] Start()
0625 03:44:07.906738 (+ 31us) write_op.cc:276] Timestamp: P: 1750823019230477 usec, L: 0
0625 03:44:07.906902 (+ 164us) write_op.cc:270] Start()
0625 03:44:07.906920 (+ 18us) write_op.cc:276] Timestamp: P: 1750823019259220 usec, L: 0
0625 03:44:07.907037 (+ 117us) write_op.cc:270] Start()
0625 03:44:07.907053 (+ 16us) write_op.cc:276] Timestamp: P: 1750823019264814 usec, L: 0
0625 03:44:07.907174 (+ 121us) write_op.cc:270] Start()
0625 03:44:07.907191 (+ 17us) write_op.cc:276] Timestamp: P: 1750823019298740 usec, L: 0
0625 03:44:07.907321 (+ 130us) write_op.cc:270] Start()
0625 03:44:07.907336 (+ 15us) write_op.cc:276] Timestamp: P: 1750823019350338 usec, L: 0
0625 03:44:07.907446 (+ 110us) write_op.cc:270] Start()
0625 03:44:07.907461 (+ 15us) write_op.cc:276] Timestamp: P: 1750823019362148 usec, L: 0
0625 03:44:07.907658 (+ 197us) write_op.cc:270] Start()
0625 03:44:07.907678 (+ 20us) write_op.cc:276] Timestamp: P: 1750823019367988 usec, L: 0
0625 03:44:07.907905 (+ 227us) write_op.cc:270] Start()
0625 03:44:07.907928 (+ 23us) write_op.cc:276] Timestamp: P: 1750823019427442 usec, L: 0
0625 03:44:07.908074 (+ 146us) write_op.cc:270] Start()
0625 03:44:07.908098 (+ 24us) write_op.cc:276] Timestamp: P: 1750823019442026 usec, L: 0
0625 03:44:07.908246 (+ 148us) write_op.cc:270] Start()
0625 03:44:07.908264 (+ 18us) write_op.cc:276] Timestamp: P: 1750823019483061 usec, L: 0
0625 03:44:07.908441 (+ 177us) write_op.cc:270] Start()
0625 03:44:07.908458 (+ 17us) write_op.cc:276] Timestamp: P: 1750823019502800 usec, L: 0
0625 03:44:07.908592 (+ 134us) write_op.cc:270] Start()
0625 03:44:07.908608 (+ 16us) write_op.cc:276] Timestamp: P: 1750823019504345 usec, L: 0
0625 03:44:07.908714 (+ 106us) write_op.cc:270] Start()
0625 03:44:07.908729 (+ 15us) write_op.cc:276] Timestamp: P: 1750823019554032 usec, L: 0
0625 03:44:07.908827 (+ 98us) write_op.cc:270] Start()
0625 03:44:07.908841 (+ 14us) write_op.cc:276] Timestamp: P: 1750823019571291 usec, L: 0
0625 03:44:07.909190 (+ 349us) write_op.cc:270] Start()
0625 03:44:07.909218 (+ 28us) write_op.cc:276] Timestamp: P: 1750823019588329 usec, L: 0
0625 03:44:07.909498 (+ 280us) write_op.cc:270] Start()
0625 03:44:07.909521 (+ 23us) write_op.cc:276] Timestamp: P: 1750823019597172 usec, L: 0
0625 03:44:07.909767 (+ 246us) write_op.cc:270] Start()
0625 03:44:07.909789 (+ 22us) write_op.cc:276] Timestamp: P: 1750823019612423 usec, L: 0
0625 03:44:07.909941 (+ 152us) write_op.cc:270] Start()
0625 03:44:07.909963 (+ 22us) write_op.cc:276] Timestamp: P: 1750823019614130 usec, L: 0
0625 03:44:07.910255 (+ 292us) write_op.cc:270] Start()
0625 03:44:07.910377 (+ 122us) write_op.cc:276] Timestamp: P: 1750823019647199 usec, L: 0
0625 03:44:07.910689 (+ 312us) write_op.cc:270] Start()
0625 03:44:07.910717 (+ 28us) write_op.cc:276] Timestamp: P: 1750823019651312 usec, L: 0
0625 03:44:07.910966 (+ 249us) write_op.cc:270] Start()
0625 03:44:07.932331 (+ 21365us) write_op.cc:276] Timestamp: P: 1750823019665124 usec, L: 0
0625 03:44:07.933374 (+ 1043us) write_op.cc:270] Start()
0625 03:44:07.933478 (+ 104us) write_op.cc:276] Timestamp: P: 1750823019684131 usec, L: 0
0625 03:44:07.934219 (+ 741us) write_op.cc:270] Start()
0625 03:44:07.934285 (+ 66us) write_op.cc:276] Timestamp: P: 1750823019688891 usec, L: 0
0625 03:44:07.935000 (+ 715us) write_op.cc:270] Start()
0625 03:44:07.935102 (+ 102us) write_op.cc:276] Timestamp: P: 1750823019707251 usec, L: 0
0625 03:44:07.935584 (+ 482us) write_op.cc:270] Start()
0625 03:44:07.952400 (+ 16816us) write_op.cc:276] Timestamp: P: 1750823019713549 usec, L: 0
0625 03:44:07.953514 (+ 1114us) write_op.cc:270] Start()
0625 03:44:07.953612 (+ 98us) write_op.cc:276] Timestamp: P: 1750823019737442 usec, L: 0
0625 03:44:07.954181 (+ 569us) write_op.cc:270] Start()
0625 03:44:07.954242 (+ 61us) write_op.cc:276] Timestamp: P: 1750823019776754 usec, L: 0
0625 03:44:07.954991 (+ 749us) write_op.cc:270] Start()
0625 03:44:07.955168 (+ 177us) write_op.cc:276] Timestamp: P: 1750823019795698 usec, L: 0
0625 03:44:07.955402 (+ 234us) write_op.cc:270] Start()
0625 03:44:07.955425 (+ 23us) write_op.cc:276] Timestamp: P: 1750823019808833 usec, L: 0
0625 03:44:07.955756 (+ 331us) write_op.cc:270] Start()
0625 03:44:07.955782 (+ 26us) write_op.cc:276] Timestamp: P: 1750823019837112 usec, L: 0
0625 03:44:07.956015 (+ 233us) write_op.cc:270] Start()
0625 03:44:07.956038 (+ 23us) write_op.cc:276] Timestamp: P: 1750823019847037 usec, L: 0
0625 03:44:07.956223 (+ 185us) write_op.cc:270] Start()
0625 03:44:07.956245 (+ 22us) write_op.cc:276] Timestamp: P: 1750823019855930 usec, L: 0
0625 03:44:07.959871 (+ 3626us) write_op.cc:270] Start()
0625 03:44:07.959900 (+ 29us) write_op.cc:276] Timestamp: P: 1750823019865067 usec, L: 0
0625 03:44:07.960140 (+ 240us) write_op.cc:270] Start()
0625 03:44:07.960166 (+ 26us) write_op.cc:276] Timestamp: P: 1750823019886116 usec, L: 0
0625 03:44:07.960323 (+ 157us) write_op.cc:270] Start()
0625 03:44:07.960344 (+ 21us) write_op.cc:276] Timestamp: P: 1750823019887462 usec, L: 0
0625 03:44:07.960567 (+ 223us) write_op.cc:270] Start()
0625 03:44:07.960589 (+ 22us) write_op.cc:276] Timestamp: P: 1750823019908335 usec, L: 0
0625 03:44:07.960765 (+ 176us) write_op.cc:270] Start()
0625 03:44:07.960786 (+ 21us) write_op.cc:276] Timestamp: P: 1750823019925413 usec, L: 0
0625 03:44:07.960930 (+ 144us) write_op.cc:270] Start()
0625 03:44:07.960950 (+ 20us) write_op.cc:276] Timestamp: P: 1750823019926816 usec, L: 0
0625 03:44:07.961107 (+ 157us) write_op.cc:270] Start()
0625 03:44:07.961127 (+ 20us) write_op.cc:276] Timestamp: P: 1750823019962471 usec, L: 0
0625 03:44:07.961304 (+ 177us) write_op.cc:270] Start()
0625 03:44:07.961324 (+ 20us) write_op.cc:276] Timestamp: P: 1750823019972055 usec, L: 0
0625 03:44:07.961458 (+ 134us) write_op.cc:270] Start()
0625 03:44:07.961478 (+ 20us) write_op.cc:276] Timestamp: P: 1750823019973374 usec, L: 0
0625 03:44:07.961657 (+ 179us) write_op.cc:270] Start()
0625 03:44:07.961677 (+ 20us) write_op.cc:276] Timestamp: P: 1750823019995762 usec, L: 0
0625 03:44:07.961803 (+ 126us) write_op.cc:270] Start()
0625 03:44:07.961821 (+ 18us) write_op.cc:276] Timestamp: P: 1750823020008579 usec, L: 0
0625 03:44:07.961977 (+ 156us) write_op.cc:270] Start()
0625 03:44:07.962023 (+ 46us) write_op.cc:276] Timestamp: P: 1750823020022083 usec, L: 0
0625 03:44:07.962214 (+ 191us) write_op.cc:270] Start()
0625 03:44:07.962243 (+ 29us) write_op.cc:276] Timestamp: P: 1750823020029242 usec, L: 0
0625 03:44:07.962446 (+ 203us) write_op.cc:270] Start()
0625 03:44:07.962468 (+ 22us) write_op.cc:276] Timestamp: P: 1750823020054769 usec, L: 0
0625 03:44:07.962621 (+ 153us) write_op.cc:270] Start()
0625 03:44:07.962642 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020070950 usec, L: 0
0625 03:44:07.962793 (+ 151us) write_op.cc:270] Start()
0625 03:44:07.962813 (+ 20us) write_op.cc:276] Timestamp: P: 1750823020097689 usec, L: 0
0625 03:44:07.962972 (+ 159us) write_op.cc:270] Start()
0625 03:44:07.962992 (+ 20us) write_op.cc:276] Timestamp: P: 1750823020127139 usec, L: 0
0625 03:44:07.963127 (+ 135us) write_op.cc:270] Start()
0625 03:44:07.963145 (+ 18us) write_op.cc:276] Timestamp: P: 1750823020144471 usec, L: 0
0625 03:44:07.968365 (+ 5220us) write_op.cc:270] Start()
0625 03:44:07.968401 (+ 36us) write_op.cc:276] Timestamp: P: 1750823020154876 usec, L: 0
0625 03:44:07.968653 (+ 252us) write_op.cc:270] Start()
0625 03:44:07.968674 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020183420 usec, L: 0
0625 03:44:07.968842 (+ 168us) write_op.cc:270] Start()
0625 03:44:07.968865 (+ 23us) write_op.cc:276] Timestamp: P: 1750823020192453 usec, L: 0
0625 03:44:07.968999 (+ 134us) write_op.cc:270] Start()
0625 03:44:07.969029 (+ 30us) write_op.cc:276] Timestamp: P: 1750823020197397 usec, L: 0
0625 03:44:07.969174 (+ 145us) write_op.cc:270] Start()
0625 03:44:07.969194 (+ 20us) write_op.cc:276] Timestamp: P: 1750823020213972 usec, L: 0
0625 03:44:07.969337 (+ 143us) write_op.cc:270] Start()
0625 03:44:07.969356 (+ 19us) write_op.cc:276] Timestamp: P: 1750823020245524 usec, L: 0
0625 03:44:07.969472 (+ 116us) write_op.cc:270] Start()
0625 03:44:07.969489 (+ 17us) write_op.cc:276] Timestamp: P: 1750823020248330 usec, L: 0
0625 03:44:07.969664 (+ 175us) write_op.cc:270] Start()
0625 03:44:07.969685 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020274231 usec, L: 0
0625 03:44:07.969878 (+ 193us) write_op.cc:270] Start()
0625 03:44:07.969900 (+ 22us) write_op.cc:276] Timestamp: P: 1750823020288195 usec, L: 0
0625 03:44:07.970100 (+ 200us) write_op.cc:270] Start()
0625 03:44:07.970121 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020309201 usec, L: 0
0625 03:44:07.970302 (+ 181us) write_op.cc:270] Start()
0625 03:44:07.970324 (+ 22us) write_op.cc:276] Timestamp: P: 1750823020323697 usec, L: 0
0625 03:44:07.970464 (+ 140us) write_op.cc:270] Start()
0625 03:44:07.970485 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020349890 usec, L: 0
0625 03:44:07.970628 (+ 143us) write_op.cc:270] Start()
0625 03:44:07.970649 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020369914 usec, L: 0
0625 03:44:07.970784 (+ 135us) write_op.cc:270] Start()
0625 03:44:07.970805 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020376396 usec, L: 0
0625 03:44:07.970960 (+ 155us) write_op.cc:270] Start()
0625 03:44:07.970979 (+ 19us) write_op.cc:276] Timestamp: P: 1750823020397987 usec, L: 0
0625 03:44:07.971124 (+ 145us) write_op.cc:270] Start()
0625 03:44:07.971146 (+ 22us) write_op.cc:276] Timestamp: P: 1750823020434367 usec, L: 0
0625 03:44:07.971314 (+ 168us) write_op.cc:270] Start()
0625 03:44:07.971335 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020444186 usec, L: 0
0625 03:44:07.971474 (+ 139us) write_op.cc:270] Start()
0625 03:44:07.971493 (+ 19us) write_op.cc:276] Timestamp: P: 1750823020466655 usec, L: 0
0625 03:44:07.979885 (+ 8392us) write_op.cc:270] Start()
0625 03:44:07.979925 (+ 40us) write_op.cc:276] Timestamp: P: 1750823020484801 usec, L: 0
0625 03:44:07.980266 (+ 341us) write_op.cc:270] Start()
0625 03:44:07.980362 (+ 96us) write_op.cc:276] Timestamp: P: 1750823020500997 usec, L: 0
0625 03:44:07.980682 (+ 320us) write_op.cc:270] Start()
0625 03:44:07.980706 (+ 24us) write_op.cc:276] Timestamp: P: 1750823020529838 usec, L: 0
0625 03:44:07.980893 (+ 187us) write_op.cc:270] Start()
0625 03:44:07.980913 (+ 20us) write_op.cc:276] Timestamp: P: 1750823020556802 usec, L: 0
0625 03:44:07.981113 (+ 200us) write_op.cc:270] Start()
0625 03:44:07.981134 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020563044 usec, L: 0
0625 03:44:07.981270 (+ 136us) write_op.cc:270] Start()
0625 03:44:07.981291 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020580391 usec, L: 0
0625 03:44:07.981479 (+ 188us) write_op.cc:270] Start()
0625 03:44:07.981500 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020594992 usec, L: 0
0625 03:44:07.981644 (+ 144us) write_op.cc:270] Start()
0625 03:44:07.981663 (+ 19us) write_op.cc:276] Timestamp: P: 1750823020616249 usec, L: 0
0625 03:44:07.981973 (+ 310us) write_op.cc:270] Start()
0625 03:44:07.982020 (+ 47us) write_op.cc:276] Timestamp: P: 1750823020631902 usec, L: 0
0625 03:44:07.982272 (+ 252us) write_op.cc:270] Start()
0625 03:44:07.982296 (+ 24us) write_op.cc:276] Timestamp: P: 1750823020668661 usec, L: 0
0625 03:44:07.982447 (+ 151us) write_op.cc:270] Start()
0625 03:44:07.982466 (+ 19us) write_op.cc:276] Timestamp: P: 1750823020672924 usec, L: 0
0625 03:44:07.985341 (+ 2875us) write_op.cc:270] Start()
0625 03:44:07.985368 (+ 27us) write_op.cc:276] Timestamp: P: 1750823020683145 usec, L: 0
0625 03:44:07.985591 (+ 223us) write_op.cc:270] Start()
0625 03:44:07.985615 (+ 24us) write_op.cc:276] Timestamp: P: 1750823020705191 usec, L: 0
0625 03:44:07.985843 (+ 228us) write_op.cc:270] Start()
0625 03:44:07.985869 (+ 26us) write_op.cc:276] Timestamp: P: 1750823020727965 usec, L: 0
0625 03:44:07.986802 (+ 933us) write_op.cc:270] Start()
0625 03:44:07.986829 (+ 27us) write_op.cc:276] Timestamp: P: 1750823020766677 usec, L: 0
0625 03:44:07.987061 (+ 232us) write_op.cc:270] Start()
0625 03:44:07.987085 (+ 24us) write_op.cc:276] Timestamp: P: 1750823020778352 usec, L: 0
0625 03:44:07.987279 (+ 194us) write_op.cc:270] Start()
0625 03:44:07.987303 (+ 24us) write_op.cc:276] Timestamp: P: 1750823020789672 usec, L: 0
0625 03:44:07.987567 (+ 264us) write_op.cc:270] Start()
0625 03:44:07.987597 (+ 30us) write_op.cc:276] Timestamp: P: 1750823020835756 usec, L: 0
0625 03:44:07.987792 (+ 195us) write_op.cc:270] Start()
0625 03:44:07.987814 (+ 22us) write_op.cc:276] Timestamp: P: 1750823020848150 usec, L: 0
0625 03:44:07.988048 (+ 234us) write_op.cc:270] Start()
0625 03:44:07.988070 (+ 22us) write_op.cc:276] Timestamp: P: 1750823020862131 usec, L: 0
0625 03:44:07.988254 (+ 184us) write_op.cc:270] Start()
0625 03:44:07.988273 (+ 19us) write_op.cc:276] Timestamp: P: 1750823020872941 usec, L: 0
0625 03:44:07.988434 (+ 161us) write_op.cc:270] Start()
0625 03:44:07.988453 (+ 19us) write_op.cc:276] Timestamp: P: 1750823020913519 usec, L: 0
0625 03:44:07.988652 (+ 199us) write_op.cc:270] Start()
0625 03:44:07.988674 (+ 22us) write_op.cc:276] Timestamp: P: 1750823020934438 usec, L: 0
0625 03:44:07.988884 (+ 210us) write_op.cc:270] Start()
0625 03:44:07.988917 (+ 33us) write_op.cc:276] Timestamp: P: 1750823020949433 usec, L: 0
0625 03:44:07.989145 (+ 228us) write_op.cc:270] Start()
0625 03:44:07.989166 (+ 21us) write_op.cc:276] Timestamp: P: 1750823020974034 usec, L: 0
0625 03:44:07.989348 (+ 182us) write_op.cc:270] Start()
0625 03:44:07.989370 (+ 22us) write_op.cc:276] Timestamp: P: 1750823020990718 usec, L: 0
0625 03:44:08.038627 (+ 49257us) write_op.cc:270] Start()
0625 03:44:08.038682 (+ 55us) write_op.cc:276] Timestamp: P: 1750823021019534 usec, L: 0
0625 03:44:08.038991 (+ 309us) write_op.cc:270] Start()
0625 03:44:08.039038 (+ 47us) write_op.cc:276] Timestamp: P: 1750823021027025 usec, L: 0
0625 03:44:08.039233 (+ 195us) write_op.cc:270] Start()
0625 03:44:08.039255 (+ 22us) write_op.cc:276] Timestamp: P: 1750823021045091 usec, L: 0
0625 03:44:08.039406 (+ 151us) write_op.cc:270] Start()
0625 03:44:08.039425 (+ 19us) write_op.cc:276] Timestamp: P: 1750823021062017 usec, L: 0
0625 03:44:08.039578 (+ 153us) write_op.cc:270] Start()
0625 03:44:08.039599 (+ 21us) write_op.cc:276] Timestamp: P: 1750823021075175 usec, L: 0
0625 03:44:08.061209 (+ 21610us) write_op.cc:270] Start()
0625 03:44:08.061263 (+ 54us) write_op.cc:276] Timestamp: P: 1750823021112548 usec, L: 0
0625 03:44:08.061540 (+ 277us) write_op.cc:270] Start()
0625 03:44:08.061569 (+ 29us) write_op.cc:276] Timestamp: P: 1750823021136097 usec, L: 0
0625 03:44:08.061765 (+ 196us) write_op.cc:270] Start()
0625 03:44:08.061787 (+ 22us) write_op.cc:276] Timestamp: P: 1750823021140024 usec, L: 0
0625 03:44:08.061949 (+ 162us) write_op.cc:270] Start()
0625 03:44:08.061971 (+ 22us) write_op.cc:276] Timestamp: P: 1750823021184216 usec, L: 0
0625 03:44:08.062137 (+ 166us) write_op.cc:270] Start()
0625 03:44:08.062160 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021207745 usec, L: 0
0625 03:44:08.084579 (+ 22419us) write_op.cc:270] Start()
0625 03:44:08.084779 (+ 200us) write_op.cc:276] Timestamp: P: 1750823021234228 usec, L: 0
0625 03:44:08.085314 (+ 535us) write_op.cc:270] Start()
0625 03:44:08.085337 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021241956 usec, L: 0
0625 03:44:08.085519 (+ 182us) write_op.cc:270] Start()
0625 03:44:08.085559 (+ 40us) write_op.cc:276] Timestamp: P: 1750823021253947 usec, L: 0
0625 03:44:08.085761 (+ 202us) write_op.cc:270] Start()
0625 03:44:08.085790 (+ 29us) write_op.cc:276] Timestamp: P: 1750823021285049 usec, L: 0
0625 03:44:08.086159 (+ 369us) write_op.cc:270] Start()
0625 03:44:08.086288 (+ 129us) write_op.cc:276] Timestamp: P: 1750823021295012 usec, L: 0
0625 03:44:08.086864 (+ 576us) write_op.cc:270] Start()
0625 03:44:08.086907 (+ 43us) write_op.cc:276] Timestamp: P: 1750823021310130 usec, L: 0
0625 03:44:08.087369 (+ 462us) write_op.cc:270] Start()
0625 03:44:08.087416 (+ 47us) write_op.cc:276] Timestamp: P: 1750823021333389 usec, L: 0
0625 03:44:08.103820 (+ 16404us) write_op.cc:270] Start()
0625 03:44:08.103865 (+ 45us) write_op.cc:276] Timestamp: P: 1750823021334792 usec, L: 0
0625 03:44:08.104771 (+ 906us) write_op.cc:270] Start()
0625 03:44:08.111613 (+ 6842us) write_op.cc:276] Timestamp: P: 1750823021363506 usec, L: 0
0625 03:44:08.112103 (+ 490us) write_op.cc:270] Start()
0625 03:44:08.112141 (+ 38us) write_op.cc:276] Timestamp: P: 1750823021391055 usec, L: 0
0625 03:44:08.112383 (+ 242us) write_op.cc:270] Start()
0625 03:44:08.112410 (+ 27us) write_op.cc:276] Timestamp: P: 1750823021406901 usec, L: 0
0625 03:44:08.112650 (+ 240us) write_op.cc:270] Start()
0625 03:44:08.112681 (+ 31us) write_op.cc:276] Timestamp: P: 1750823021408823 usec, L: 0
0625 03:44:08.112948 (+ 267us) write_op.cc:270] Start()
0625 03:44:08.112974 (+ 26us) write_op.cc:276] Timestamp: P: 1750823021443376 usec, L: 0
0625 03:44:08.113308 (+ 334us) write_op.cc:270] Start()
0625 03:44:08.119775 (+ 6467us) write_op.cc:276] Timestamp: P: 1750823021458231 usec, L: 0
0625 03:44:08.120157 (+ 382us) write_op.cc:270] Start()
0625 03:44:08.120192 (+ 35us) write_op.cc:276] Timestamp: P: 1750823021469039 usec, L: 0
0625 03:44:08.120435 (+ 243us) write_op.cc:270] Start()
0625 03:44:08.120463 (+ 28us) write_op.cc:276] Timestamp: P: 1750823021473256 usec, L: 0
0625 03:44:08.120637 (+ 174us) write_op.cc:270] Start()
0625 03:44:08.120662 (+ 25us) write_op.cc:276] Timestamp: P: 1750823021500349 usec, L: 0
0625 03:44:08.120861 (+ 199us) write_op.cc:270] Start()
0625 03:44:08.120887 (+ 26us) write_op.cc:276] Timestamp: P: 1750823021508458 usec, L: 0
0625 03:44:08.121215 (+ 328us) write_op.cc:270] Start()
0625 03:44:08.121270 (+ 55us) write_op.cc:276] Timestamp: P: 1750823021509982 usec, L: 0
0625 03:44:08.121622 (+ 352us) write_op.cc:270] Start()
0625 03:44:08.121656 (+ 34us) write_op.cc:276] Timestamp: P: 1750823021537337 usec, L: 0
0625 03:44:08.121949 (+ 293us) write_op.cc:270] Start()
0625 03:44:08.121996 (+ 47us) write_op.cc:276] Timestamp: P: 1750823021564407 usec, L: 0
0625 03:44:08.138373 (+ 16377us) write_op.cc:270] Start()
0625 03:44:08.138429 (+ 56us) write_op.cc:276] Timestamp: P: 1750823021570454 usec, L: 0
0625 03:44:08.138745 (+ 316us) write_op.cc:270] Start()
0625 03:44:08.138775 (+ 30us) write_op.cc:276] Timestamp: P: 1750823021584975 usec, L: 0
0625 03:44:08.138985 (+ 210us) write_op.cc:270] Start()
0625 03:44:08.139008 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021597204 usec, L: 0
0625 03:44:08.139229 (+ 221us) write_op.cc:270] Start()
0625 03:44:08.139253 (+ 24us) write_op.cc:276] Timestamp: P: 1750823021611967 usec, L: 0
0625 03:44:08.139461 (+ 208us) write_op.cc:270] Start()
0625 03:44:08.139483 (+ 22us) write_op.cc:276] Timestamp: P: 1750823021621537 usec, L: 0
0625 03:44:08.139640 (+ 157us) write_op.cc:270] Start()
0625 03:44:08.139662 (+ 22us) write_op.cc:276] Timestamp: P: 1750823021669003 usec, L: 0
0625 03:44:08.139896 (+ 234us) write_op.cc:270] Start()
0625 03:44:08.140023 (+ 127us) write_op.cc:276] Timestamp: P: 1750823021681549 usec, L: 0
0625 03:44:08.140291 (+ 268us) write_op.cc:270] Start()
0625 03:44:08.151623 (+ 11332us) write_op.cc:276] Timestamp: P: 1750823021683180 usec, L: 0
0625 03:44:08.153305 (+ 1682us) write_op.cc:270] Start()
0625 03:44:08.153338 (+ 33us) write_op.cc:276] Timestamp: P: 1750823021722292 usec, L: 0
0625 03:44:08.153879 (+ 541us) write_op.cc:270] Start()
0625 03:44:08.153907 (+ 28us) write_op.cc:276] Timestamp: P: 1750823021736548 usec, L: 0
0625 03:44:08.154501 (+ 594us) write_op.cc:270] Start()
0625 03:44:08.154524 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021742111 usec, L: 0
0625 03:44:08.159760 (+ 5236us) write_op.cc:270] Start()
0625 03:44:08.159794 (+ 34us) write_op.cc:276] Timestamp: P: 1750823021754228 usec, L: 0
0625 03:44:08.160040 (+ 246us) write_op.cc:270] Start()
0625 03:44:08.160063 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021789017 usec, L: 0
0625 03:44:08.160224 (+ 161us) write_op.cc:270] Start()
0625 03:44:08.160247 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021833347 usec, L: 0
0625 03:44:08.160459 (+ 212us) write_op.cc:270] Start()
0625 03:44:08.160483 (+ 24us) write_op.cc:276] Timestamp: P: 1750823021856737 usec, L: 0
0625 03:44:08.160730 (+ 247us) write_op.cc:270] Start()
0625 03:44:08.160753 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021864398 usec, L: 0
0625 03:44:08.160918 (+ 165us) write_op.cc:270] Start()
0625 03:44:08.160939 (+ 21us) write_op.cc:276] Timestamp: P: 1750823021878715 usec, L: 0
0625 03:44:08.161102 (+ 163us) write_op.cc:270] Start()
0625 03:44:08.161125 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021907195 usec, L: 0
0625 03:44:08.161314 (+ 189us) write_op.cc:270] Start()
0625 03:44:08.166239 (+ 4925us) write_op.cc:276] Timestamp: P: 1750823021930487 usec, L: 0
0625 03:44:08.168808 (+ 2569us) write_op.cc:270] Start()
0625 03:44:08.168845 (+ 37us) write_op.cc:276] Timestamp: P: 1750823021960234 usec, L: 0
0625 03:44:08.169128 (+ 283us) write_op.cc:270] Start()
0625 03:44:08.169151 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021971958 usec, L: 0
0625 03:44:08.169351 (+ 200us) write_op.cc:270] Start()
0625 03:44:08.169374 (+ 23us) write_op.cc:276] Timestamp: P: 1750823021981178 usec, L: 0
0625 03:44:08.169510 (+ 136us) write_op.cc:270] Start()
0625 03:44:08.169528 (+ 18us) write_op.cc:276] Timestamp: P: 1750823021998368 usec, L: 0
0625 03:44:08.169676 (+ 148us) write_op.cc:270] Start()
0625 03:44:08.169696 (+ 20us) write_op.cc:276] Timestamp: P: 1750823022021477 usec, L: 0
0625 03:44:08.169846 (+ 150us) write_op.cc:270] Start()
0625 03:44:08.171060 (+ 1214us) write_op.cc:276] Timestamp: P: 1750823022054753 usec, L: 0
0625 03:44:08.171293 (+ 233us) write_op.cc:270] Start()
0625 03:44:08.171317 (+ 24us) write_op.cc:276] Timestamp: P: 1750823022056381 usec, L: 0
0625 03:44:08.171459 (+ 142us) write_op.cc:270] Start()
0625 03:44:08.171478 (+ 19us) write_op.cc:276] Timestamp: P: 1750823022069291 usec, L: 0
0625 03:44:08.171628 (+ 150us) write_op.cc:270] Start()
0625 03:44:08.171644 (+ 16us) write_op.cc:276] Timestamp: P: 1750823022120214 usec, L: 0
0625 03:44:08.171802 (+ 158us) write_op.cc:270] Start()
0625 03:44:08.171822 (+ 20us) write_op.cc:276] Timestamp: P: 1750823022124887 usec, L: 0
0625 03:44:08.171981 (+ 159us) write_op.cc:270] Start()
0625 03:44:08.172032 (+ 51us) write_op.cc:276] Timestamp: P: 1750823022182362 usec, L: 0
0625 03:44:08.172479 (+ 447us) write_op.cc:270] Start()
0625 03:44:08.172507 (+ 28us) write_op.cc:276] Timestamp: P: 1750823022235008 usec, L: 0
0625 03:44:08.173027 (+ 520us) write_op.cc:270] Start()
0625 03:44:08.173052 (+ 25us) write_op.cc:276] Timestamp: P: 1750823022257488 usec, L: 0
0625 03:44:08.173234 (+ 182us) write_op.cc:270] Start()
0625 03:44:08.173255 (+ 21us) write_op.cc:276] Timestamp: P: 1750823022280169 usec, L: 0
0625 03:44:08.173397 (+ 142us) write_op.cc:270] Start()
0625 03:44:08.173419 (+ 22us) write_op.cc:276] Timestamp: P: 1750823022295232 usec, L: 0
0625 03:44:08.173564 (+ 145us) write_op.cc:270] Start()
0625 03:44:08.173585 (+ 21us) write_op.cc:276] Timestamp: P: 1750823022332654 usec, L: 0
0625 03:44:08.173726 (+ 141us) write_op.cc:270] Start()
0625 03:44:08.173746 (+ 20us) write_op.cc:276] Timestamp: P: 1750823022353973 usec, L: 0
0625 03:44:08.173904 (+ 158us) write_op.cc:270] Start()
0625 03:44:08.173925 (+ 21us) write_op.cc:276] Timestamp: P: 17508
I20250625 03:44:09.602485 15227 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:44:09.603520 15227 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:44:09.609349 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2043 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:44:09.616972 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2043 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:44:09.617120 14752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2043 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:44:09.620916 14840 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:44:09.650041 14134 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2043 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:44:09.627111 15227 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:44:09.643134 15253 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:44:10.959486 15253 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:44:10.959883 15253 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:44:10.961853 15253 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:44:10.967164 14753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2468 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:44:10.980306 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2468 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:44:10.980260 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2468 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:44:10.984143 13698 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2468 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:44:10.984742 14838 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:44:10.985667 15253 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:44:10.997579 14906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.12.175.129:45479 (request call id 109) took 1571 ms. Trace:
I20250625 03:44:11.187841 15262 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:44:11.188474 15262 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:44:11.196024 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 2698 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:44:11.200618 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 2698 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:44:11.201918 15262 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:44:11.203648 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 2698 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:44:11.204530 14684 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:44:11.204790 14905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 2698 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:44:11.205762 15262 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:44:11.205760 14905 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 4.
I20250625 03:44:11.557736 15266 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:44:11.558216 15266 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:44:11.562564 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2468 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:44:11.564316 13697 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2468 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:44:11.565265 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2468 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:44:11.566834 14752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2468 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:44:11.567039 14841 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:44:10.999271 14906 rpcz_store.cc:276] 0625 03:44:09.426470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:44:09.426542 (+ 72us) service_pool.cc:225] Handling call
0625 03:44:09.847176 (+420634us) raft_consensus.cc:1469] Updating replica for 425 ops
0625 03:44:10.268271 (+421095us) raft_consensus.cc:1532] Early marking committed up to index 0
0625 03:44:10.268277 (+ 6us) raft_consensus.cc:1537] Triggering prepare for 425 ops
0625 03:44:10.295369 (+ 27092us) write_op.cc:270] Start()
0625 03:44:10.295443 (+ 74us) write_op.cc:276] Timestamp: P: 1750823028339230 usec, L: 0
0625 03:44:10.296061 (+ 618us) write_op.cc:270] Start()
0625 03:44:10.296090 (+ 29us) write_op.cc:276] Timestamp: P: 1750823028389504 usec, L: 0
0625 03:44:10.296391 (+ 301us) write_op.cc:270] Start()
0625 03:44:10.296420 (+ 29us) write_op.cc:276] Timestamp: P: 1750823028434301 usec, L: 0
0625 03:44:10.296684 (+ 264us) write_op.cc:270] Start()
0625 03:44:10.296708 (+ 24us) write_op.cc:276] Timestamp: P: 1750823028441910 usec, L: 0
0625 03:44:10.297058 (+ 350us) write_op.cc:270] Start()
0625 03:44:10.297084 (+ 26us) write_op.cc:276] Timestamp: P: 1750823028542248 usec, L: 0
0625 03:44:10.298094 (+ 1010us) write_op.cc:270] Start()
0625 03:44:10.298118 (+ 24us) write_op.cc:276] Timestamp: P: 1750823028635588 usec, L: 0
0625 03:44:10.298364 (+ 246us) write_op.cc:270] Start()
0625 03:44:10.298428 (+ 64us) write_op.cc:276] Timestamp: P: 1750823028773375 usec, L: 0
0625 03:44:10.298626 (+ 198us) write_op.cc:270] Start()
0625 03:44:10.298651 (+ 25us) write_op.cc:276] Timestamp: P: 1750823028807373 usec, L: 0
0625 03:44:10.299344 (+ 693us) write_op.cc:270] Start()
0625 03:44:10.299372 (+ 28us) write_op.cc:276] Timestamp: P: 1750823028843762 usec, L: 0
0625 03:44:10.299588 (+ 216us) write_op.cc:270] Start()
0625 03:44:10.299611 (+ 23us) write_op.cc:276] Timestamp: P: 1750823028888711 usec, L: 0
0625 03:44:10.299822 (+ 211us) write_op.cc:270] Start()
0625 03:44:10.299861 (+ 39us) write_op.cc:276] Timestamp: P: 1750823028893775 usec, L: 0
0625 03:44:10.300066 (+ 205us) write_op.cc:270] Start()
0625 03:44:10.300088 (+ 22us) write_op.cc:276] Timestamp: P: 1750823028930134 usec, L: 0
0625 03:44:10.300322 (+ 234us) write_op.cc:270] Start()
0625 03:44:10.300345 (+ 23us) write_op.cc:276] Timestamp: P: 1750823029035619 usec, L: 0
0625 03:44:10.300928 (+ 583us) write_op.cc:270] Start()
0625 03:44:10.300954 (+ 26us) write_op.cc:276] Timestamp: P: 1750823029117090 usec, L: 0
0625 03:44:10.301134 (+ 180us) write_op.cc:270] Start()
0625 03:44:10.301156 (+ 22us) write_op.cc:276] Timestamp: P: 1750823029200663 usec, L: 0
0625 03:44:10.301357 (+ 201us) write_op.cc:270] Start()
0625 03:44:10.301378 (+ 21us) write_op.cc:276] Timestamp: P: 1750823029225117 usec, L: 0
0625 03:44:10.301605 (+ 227us) write_op.cc:270] Start()
0625 03:44:10.301630 (+ 25us) write_op.cc:276] Timestamp: P: 1750823029367911 usec, L: 0
0625 03:44:10.301778 (+ 148us) write_op.cc:270] Start()
0625 03:44:10.301797 (+ 19us) write_op.cc:276] Timestamp: P: 1750823029462468 usec, L: 0
0625 03:44:10.301920 (+ 123us) write_op.cc:270] Start()
0625 03:44:10.301937 (+ 17us) write_op.cc:276] Timestamp: P: 1750823029498230 usec, L: 0
0625 03:44:10.302235 (+ 298us) write_op.cc:270] Start()
0625 03:44:10.302258 (+ 23us) write_op.cc:276] Timestamp: P: 1750823029600585 usec, L: 0
0625 03:44:10.302446 (+ 188us) write_op.cc:270] Start()
0625 03:44:10.302468 (+ 22us) write_op.cc:276] Timestamp: P: 1750823029716357 usec, L: 0
0625 03:44:10.302728 (+ 260us) write_op.cc:270] Start()
0625 03:44:10.302752 (+ 24us) write_op.cc:276] Timestamp: P: 1750823029779340 usec, L: 0
0625 03:44:10.303020 (+ 268us) write_op.cc:270] Start()
0625 03:44:10.303046 (+ 26us) write_op.cc:276] Timestamp: P: 1750823029857160 usec, L: 0
0625 03:44:10.303260 (+ 214us) write_op.cc:270] Start()
0625 03:44:10.303285 (+ 25us) write_op.cc:276] Timestamp: P: 1750823029904852 usec, L: 0
0625 03:44:10.303591 (+ 306us) write_op.cc:270] Start()
0625 03:44:10.303619 (+ 28us) write_op.cc:276] Timestamp: P: 1750823029942233 usec, L: 0
0625 03:44:10.303866 (+ 247us) write_op.cc:270] Start()
0625 03:44:10.303889 (+ 23us) write_op.cc:276] Timestamp: P: 1750823030001471 usec, L: 0
0625 03:44:10.304071 (+ 182us) write_op.cc:270] Start()
0625 03:44:10.304091 (+ 20us) write_op.cc:276] Timestamp: P: 1750823030002774 usec, L: 0
0625 03:44:10.307460 (+ 3369us) write_op.cc:270] Start()
0625 03:44:10.307488 (+ 28us) write_op.cc:276] Timestamp: P: 1750823030015816 usec, L: 0
0625 03:44:10.310125 (+ 2637us) write_op.cc:270] Start()
0625 03:44:10.310154 (+ 29us) write_op.cc:276] Timestamp: P: 1750823030064606 usec, L: 0
0625 03:44:10.310349 (+ 195us) write_op.cc:270] Start()
0625 03:44:10.310372 (+ 23us) write_op.cc:276] Timestamp: P: 1750823030104739 usec, L: 0
0625 03:44:10.310531 (+ 159us) write_op.cc:270] Start()
0625 03:44:10.310551 (+ 20us) write_op.cc:276] Timestamp: P: 1750823030106217 usec, L: 0
0625 03:44:10.311236 (+ 685us) write_op.cc:270] Start()
0625 03:44:10.311262 (+ 26us) write_op.cc:276] Timestamp: P: 1750823030232743 usec, L: 0
0625 03:44:10.311521 (+ 259us) write_op.cc:270] Start()
0625 03:44:10.311545 (+ 24us) write_op.cc:276] Timestamp: P: 1750823030268752 usec, L: 0
0625 03:44:10.311826 (+ 281us) write_op.cc:270] Start()
0625 03:44:10.311847 (+ 21us) write_op.cc:276] Timestamp: P: 1750823030306706 usec, L: 0
0625 03:44:10.312101 (+ 254us) write_op.cc:270] Start()
0625 03:44:10.312124 (+ 23us) write_op.cc:276] Timestamp: P: 1750823030344461 usec, L: 0
0625 03:44:10.312276 (+ 152us) write_op.cc:270] Start()
0625 03:44:10.312299 (+ 23us) write_op.cc:276] Timestamp: P: 1750823030409359 usec, L: 0
0625 03:44:10.319640 (+ 7341us) write_op.cc:270] Start()
0625 03:44:10.319675 (+ 35us) write_op.cc:276] Timestamp: P: 1750823030427164 usec, L: 0
0625 03:44:10.320319 (+ 644us) write_op.cc:270] Start()
0625 03:44:10.320347 (+ 28us) write_op.cc:276] Timestamp: P: 1750823030439800 usec, L: 0
0625 03:44:10.320548 (+ 201us) write_op.cc:270] Start()
0625 03:44:10.320569 (+ 21us) write_op.cc:276] Timestamp: P: 1750823030472365 usec, L: 0
0625 03:44:10.320719 (+ 150us) write_op.cc:270] Start()
0625 03:44:10.320738 (+ 19us) write_op.cc:276] Timestamp: P: 1750823030476329 usec, L: 0
0625 03:44:10.320866 (+ 128us) write_op.cc:270] Start()
0625 03:44:10.320885 (+ 19us) write_op.cc:276] Timestamp: P: 1750823030488668 usec, L: 0
0625 03:44:10.321077 (+ 192us) write_op.cc:270] Start()
0625 03:44:10.321099 (+ 22us) write_op.cc:276] Timestamp: P: 1750823030552954 usec, L: 0
0625 03:44:10.321294 (+ 195us) write_op.cc:270] Start()
0625 03:44:10.321316 (+ 22us) write_op.cc:276] Timestamp: P: 1750823030571832 usec, L: 0
0625 03:44:10.321484 (+ 168us) write_op.cc:270] Start()
0625 03:44:10.321504 (+ 20us) write_op.cc:276] Timestamp: P: 1750823030620484 usec, L: 0
0625 03:44:10.321675 (+ 171us) write_op.cc:270] Start()
0625 03:44:10.321712 (+ 37us) write_op.cc:276] Timestamp: P: 1750823030630382 usec, L: 0
0625 03:44:10.321898 (+ 186us) write_op.cc:270] Start()
0625 03:44:10.321918 (+ 20us) write_op.cc:276] Timestamp: P: 1750823030689864 usec, L: 0
0625 03:44:10.322079 (+ 161us) write_op.cc:270] Start()
0625 03:44:10.322097 (+ 18us) write_op.cc:276] Timestamp: P: 1750823030728766 usec, L: 0
0625 03:44:10.322400 (+ 303us) write_op.cc:270] Start()
0625 03:44:10.322429 (+ 29us) write_op.cc:276] Timestamp: P: 1750823030758654 usec, L: 0
0625 03:44:10.322638 (+ 209us) write_op.cc:270] Start()
0625 03:44:10.322668 (+ 30us) write_op.cc:276] Timestamp: P: 1750823030761676 usec, L: 0
0625 03:44:10.322859 (+ 191us) write_op.cc:270] Start()
0625 03:44:10.322881 (+ 22us) write_op.cc:276] Timestamp: P: 1750823030781479 usec, L: 0
0625 03:44:10.323029 (+ 148us) write_op.cc:270] Start()
0625 03:44:10.323068 (+ 39us) write_op.cc:276] Timestamp: P: 1750823030792042 usec, L: 0
0625 03:44:10.323208 (+ 140us) write_op.cc:270] Start()
0625 03:44:10.323227 (+ 19us) write_op.cc:276] Timestamp: P: 1750823030864028 usec, L: 0
0625 03:44:10.323367 (+ 140us) write_op.cc:270] Start()
0625 03:44:10.323386 (+ 19us) write_op.cc:276] Timestamp: P: 1750823030890580 usec, L: 0
0625 03:44:10.323605 (+ 219us) write_op.cc:270] Start()
0625 03:44:10.323631 (+ 26us) write_op.cc:276] Timestamp: P: 1750823030937566 usec, L: 0
0625 03:44:10.323822 (+ 191us) write_op.cc:270] Start()
0625 03:44:10.323847 (+ 25us) write_op.cc:276] Timestamp: P: 1750823030949166 usec, L: 0
0625 03:44:10.323998 (+ 151us) write_op.cc:270] Start()
0625 03:44:10.337148 (+ 13150us) write_op.cc:276] Timestamp: P: 1750823030968145 usec, L: 0
0625 03:44:10.337456 (+ 308us) write_op.cc:270] Start()
0625 03:44:10.337482 (+ 26us) write_op.cc:276] Timestamp: P: 1750823031007018 usec, L: 0
0625 03:44:10.337702 (+ 220us) write_op.cc:270] Start()
0625 03:44:10.337727 (+ 25us) write_op.cc:276] Timestamp: P: 1750823031084060 usec, L: 0
0625 03:44:10.337916 (+ 189us) write_op.cc:270] Start()
0625 03:44:10.337938 (+ 22us) write_op.cc:276] Timestamp: P: 1750823031114124 usec, L: 0
0625 03:44:10.338108 (+ 170us) write_op.cc:270] Start()
0625 03:44:10.338158 (+ 50us) write_op.cc:276] Timestamp: P: 1750823031115703 usec, L: 0
0625 03:44:10.338335 (+ 177us) write_op.cc:270] Start()
0625 03:44:10.338357 (+ 22us) write_op.cc:276] Timestamp: P: 1750823031116946 usec, L: 0
0625 03:44:10.338500 (+ 143us) write_op.cc:270] Start()
0625 03:44:10.338520 (+ 20us) write_op.cc:276] Timestamp: P: 1750823031157616 usec, L: 0
0625 03:44:10.338663 (+ 143us) write_op.cc:270] Start()
0625 03:44:10.338683 (+ 20us) write_op.cc:276] Timestamp: P: 1750823031172551 usec, L: 0
0625 03:44:10.338837 (+ 154us) write_op.cc:270] Start()
0625 03:44:10.338858 (+ 21us) write_op.cc:276] Timestamp: P: 1750823031174278 usec, L: 0
0625 03:44:10.339140 (+ 282us) write_op.cc:270] Start()
0625 03:44:10.339166 (+ 26us) write_op.cc:276] Timestamp: P: 1750823031297789 usec, L: 0
0625 03:44:10.339410 (+ 244us) write_op.cc:270] Start()
0625 03:44:10.339433 (+ 23us) write_op.cc:276] Timestamp: P: 1750823031313617 usec, L: 0
0625 03:44:10.339596 (+ 163us) write_op.cc:270] Start()
0625 03:44:10.339617 (+ 21us) write_op.cc:276] Timestamp: P: 1750823031319005 usec, L: 0
0625 03:44:10.339786 (+ 169us) write_op.cc:270] Start()
0625 03:44:10.339807 (+ 21us) write_op.cc:276] Timestamp: P: 1750823031347301 usec, L: 0
0625 03:44:10.339970 (+ 163us) write_op.cc:270] Start()
0625 03:44:10.340103 (+ 133us) write_op.cc:276] Timestamp: P: 1750823031355784 usec, L: 0
0625 03:44:10.355444 (+ 15341us) write_op.cc:270] Start()
0625 03:44:10.355496 (+ 52us) write_op.cc:276] Timestamp: P: 1750823031382335 usec, L: 0
0625 03:44:10.355900 (+ 404us) write_op.cc:270] Start()
0625 03:44:10.355928 (+ 28us) write_op.cc:276] Timestamp: P: 1750823031399200 usec, L: 0
0625 03:44:10.356129 (+ 201us) write_op.cc:270] Start()
0625 03:44:10.356154 (+ 25us) write_op.cc:276] Timestamp: P: 1750823031400560 usec, L: 0
0625 03:44:10.356312 (+ 158us) write_op.cc:270] Start()
0625 03:44:10.356335 (+ 23us) write_op.cc:276] Timestamp: P: 1750823031422831 usec, L: 0
0625 03:44:10.356511 (+ 176us) write_op.cc:270] Start()
0625 03:44:10.356534 (+ 23us) write_op.cc:276] Timestamp: P: 1750823031426364 usec, L: 0
0625 03:44:10.356706 (+ 172us) write_op.cc:270] Start()
0625 03:44:10.356727 (+ 21us) write_op.cc:276] Timestamp: P: 1750823031441789 usec, L: 0
0625 03:44:10.356888 (+ 161us) write_op.cc:270] Start()
0625 03:44:10.356909 (+ 21us) write_op.cc:276] Timestamp: P: 1750823031475402 usec, L: 0
0625 03:44:10.357079 (+ 170us) write_op.cc:270] Start()
0625 03:44:10.357100 (+ 21us) write_op.cc:276] Timestamp: P: 1750823031494575 usec, L: 0
0625 03:44:10.357267 (+ 167us) write_op.cc:270] Start()
0625 03:44:10.357288 (+ 21us) write_op.cc:276] Timestamp: P: 1750823031496805 usec, L: 0
0625 03:44:10.357426 (+ 138us) write_op.cc:270] Start()
0625 03:44:10.357444 (+ 18us) write_op.cc:276] Timestamp: P: 1750823031530044 usec, L: 0
0625 03:44:10.357613 (+ 169us) write_op.cc:270] Start()
0625 03:44:10.357633 (+ 20us) write_op.cc:276] Timestamp: P: 1750823031533823 usec, L: 0
0625 03:44:10.357786 (+ 153us) write_op.cc:270] Start()
0625 03:44:10.357805 (+ 19us) write_op.cc:276] Timestamp: P: 1750823031544716 usec, L: 0
0625 03:44:10.357953 (+ 148us) write_op.cc:270] Start()
0625 03:44:10.357971 (+ 18us) write_op.cc:276] Timestamp: P: 1750823031582284 usec, L: 0
0625 03:44:10.358348 (+ 377us) write_op.cc:270] Start()
0625 03:44:10.358387 (+ 39us) write_op.cc:276] Timestamp: P: 1750823031595321 usec, L: 0
0625 03:44:10.358544 (+ 157us) write_op.cc:270] Start()
0625 03:44:10.358564 (+ 20us) write_op.cc:276] Timestamp: P: 1750823031607001 usec, L: 0
0625 03:44:10.358873 (+ 309us) write_op.cc:270] Start()
0625 03:44:10.358896 (+ 23us) write_op.cc:276] Timestamp: P: 1750823031626809 usec, L: 0
0625 03:44:10.359065 (+ 169us) write_op.cc:270] Start()
0625 03:44:10.359092 (+ 27us) write_op.cc:276] Timestamp: P: 1750823031648558 usec, L: 0
0625 03:44:10.359285 (+ 193us) write_op.cc:270] Start()
0625 03:44:10.359308 (+ 23us) write_op.cc:276] Timestamp: P: 1750823031660610 usec, L: 0
0625 03:44:10.359481 (+ 173us) write_op.cc:270] Start()
0625 03:44:10.359500 (+ 19us) write_op.cc:276] Timestamp: P: 1750823031712133 usec, L: 0
0625 03:44:10.359650 (+ 150us) write_op.cc:270] Start()
0625 03:44:10.359679 (+ 29us) write_op.cc:276] Timestamp: P: 1750823031713695 usec, L: 0
0625 03:44:10.363815 (+ 4136us) write_op.cc:270] Start()
0625 03:44:10.363846 (+ 31us) write_op.cc:276] Timestamp: P: 1750823031715188 usec, L: 0
0625 03:44:10.364101 (+ 255us) write_op.cc:270] Start()
0625 03:44:10.364129 (+ 28us) write_op.cc:276] Timestamp: P: 1750823031739981 usec, L: 0
0625 03:44:10.364310 (+ 181us) write_op.cc:270] Start()
0625 03:44:10.364332 (+ 22us) write_op.cc:276] Timestamp: P: 1750823031764014 usec, L: 0
0625 03:44:10.401223 (+ 36891us) write_op.cc:270] Start()
0625 03:44:10.416248 (+ 15025us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x619000090858. stack: 00007f2f009d4d56 00007f2f009d4b10 00007f2eff9cb29d 0000000000970a84 0000000000970a4d 000000000097041a 00007f2f0d8e688a 00007f2f0d8ed96f 00007f2f0d66762c 00007f2f0d61f90a 00007f2f0d7dbaee 00007f2f0d7a96c7 00007f2f0d7556af 00007f2f0d75476e 00007f2f0c3f7341 00007f2f0c416a88
0625 03:44:10.416281 (+ 33us) write_op.cc:276] Timestamp: P: 1750823031771711 usec, L: 0
0625 03:44:10.416580 (+ 299us) write_op.cc:270] Start()
0625 03:44:10.416611 (+ 31us) write_op.cc:276] Timestamp: P: 1750823031799142 usec, L: 0
0625 03:44:10.416821 (+ 210us) write_op.cc:270] Start()
0625 03:44:10.416844 (+ 23us) write_op.cc:276] Timestamp: P: 1750823031802001 usec, L: 0
0625 03:44:10.417025 (+ 181us) write_op.cc:270] Start()
0625 03:44:10.417047 (+ 22us) write_op.cc:276] Timestamp: P: 1750823031809188 usec, L: 0
0625 03:44:10.417226 (+ 179us) write_op.cc:270] Start()
0625 03:44:10.417250 (+ 24us) write_op.cc:276] Timestamp: P: 1750823031837379 usec, L: 0
0625 03:44:10.417432 (+ 182us) write_op.cc:270] Start()
0625 03:44:10.417456 (+ 24us) write_op.cc:276] Timestamp: P: 1750823031877670 usec, L: 0
0625 03:44:10.417633 (+ 177us) write_op.cc:270] Start()
0625 03:44:10.417656 (+ 23us) write_op.cc:276] Timestamp: P: 1750823031900385 usec, L: 0
0625 03:44:10.417827 (+ 171us) write_op.cc:270] Start()
0625 03:44:10.417849 (+ 22us) write_op.cc:276] Timestamp: P: 1750823031908756 usec, L: 0
0625 03:44:10.417996 (+ 147us) write_op.cc:270] Start()
0625 03:44:10.418017 (+ 21us) write_op.cc:276] Timestamp: P: 1750823031945990 usec, L: 0
0625 03:44:10.418160 (+ 143us) write_op.cc:270] Start()
0625 03:44:10.418179 (+ 19us) write_op.cc:276] Timestamp: P: 1750823031986526 usec, L: 0
0625 03:44:10.418327 (+ 148us) write_op.cc:270] Start()
0625 03:44:10.418344 (+ 17us) write_op.cc:276] Timestamp: P: 1750823032010496 usec, L: 0
0625 03:44:10.418496 (+ 152us) write_op.cc:270] Start()
0625 03:44:10.418516 (+ 20us) write_op.cc:276] Timestamp: P: 1750823032039188 usec, L: 0
0625 03:44:10.419026 (+ 510us) write_op.cc:270] Start()
0625 03:44:10.419053 (+ 27us) write_op.cc:276] Timestamp: P: 1750823032040770 usec, L: 0
0625 03:44:10.419242 (+ 189us) write_op.cc:270] Start()
0625 03:44:10.419266 (+ 24us) write_op.cc:276] Timestamp: P: 1750823032104910 usec, L: 0
0625 03:44:10.419458 (+ 192us) write_op.cc:270] Start()
0625 03:44:10.419480 (+ 22us) write_op.cc:276] Timestamp: P: 1750823032137622 usec, L: 0
0625 03:44:10.419684 (+ 204us) write_op.cc:270] Start()
0625 03:44:10.419727 (+ 43us) write_op.cc:276] Timestamp: P: 1750823032166899 usec, L: 0
0625 03:44:10.419888 (+ 161us) write_op.cc:270] Start()
0625 03:44:10.419907 (+ 19us) write_op.cc:276] Timestamp: P: 1750823032168502 usec, L: 0
0625 03:44:10.420050 (+ 143us) write_op.cc:270] Start()
0625 03:44:10.420069 (+ 19us) write_op.cc:276] Timestamp: P: 1750823032177605 usec, L: 0
0625 03:44:10.420187 (+ 118us) write_op.cc:270] Start()
0625 03:44:10.420214 (+ 27us) write_op.cc:276] Timestamp: P: 1750823032219550 usec, L: 0
0625 03:44:10.420376 (+ 162us) write_op.cc:270] Start()
0625 03:44:10.420396 (+ 20us) write_op.cc:276] Timestamp: P: 1750823032237810 usec, L: 0
0625 03:44:10.420593 (+ 197us) write_op.cc:270] Start()
0625 03:44:10.420617 (+ 24us) write_op.cc:276] Timestamp: P: 1750823032278604 usec, L: 0
0625 03:44:10.420782 (+ 165us) write_op.cc:270] Start()
0625 03:44:10.420805 (+ 23us) write_op.cc:276] Timestamp: P: 1750823032297204 usec, L: 0
0625 03:44:10.420985 (+ 180us) write_op.cc:270] Start()
0625 03:44:10.421007 (+ 22us) write_op.cc:276] Timestamp: P: 1750823032299060 usec, L: 0
0625 03:44:10.421196 (+ 189us) write_op.cc:270] Start()
0625 03:44:10.421233 (+ 37us) write_op.cc:276] Timestamp: P: 1750823032341192 usec, L: 0
0625 03:44:10.421411 (+ 178us) write_op.cc:270] Start()
0625 03:44:10.421435 (+ 24us) write_op.cc:276] Timestamp: P: 1750823032370281 usec, L: 0
0625 03:44:10.421597 (+ 162us) write_op.cc:270] Start()
0625 03:44:10.421617 (+ 20us) write_op.cc:276] Timestamp: P: 1750823032393092 usec, L: 0
0625 03:44:10.421769 (+ 152us) write_op.cc:270] Start()
0625 03:44:10.421789 (+ 20us) write_op.cc:276] Timestamp: P: 1750823032411133 usec, L: 0
0625 03:44:10.421926 (+ 137us) write_op.cc:270] Start()
0625 03:44:10.421943 (+ 17us) write_op.cc:276] Timestamp: P: 1750823032452617 usec, L: 0
0625 03:44:10.422073 (+ 130us) write_op.cc:270] Start()
0625 03:44:10.422093 (+ 20us) write_op.cc:276] Timestamp: P: 1750823032455966 usec, L: 0
0625 03:44:10.422250 (+ 157us) write_op.cc:270] Start()
0625 03:44:10.422269 (+ 19us) write_op.cc:276] Timestamp: P: 1750823032457357 usec, L: 0
0625 03:44:10.422431 (+ 162us) write_op.cc:270] Start()
0625 03:44:10.422455 (+ 24us) write_op.cc:276] Timestamp: P: 1750823032493735 usec, L: 0
0625 03:44:10.422623 (+ 168us) write_op.cc:270] Start()
0625 03:44:10.422647 (+ 24us) write_op.cc:276] Timestamp: P: 1750823032575852 usec, L: 0
0625 03:44:10.422846 (+ 199us) write_op.cc:270] Start()
0625 03:44:10.422867 (+ 21us) write_op.cc:276] Timestamp: P: 1750823032577701 usec, L: 0
0625 03:44:10.423045 (+ 178us) write_op.cc:270] Start()
0625 03:44:10.423064 (+ 19us) write_op.cc:276] Timestamp: P: 1750823032620072 usec, L: 0
0625 03:44:10.423221 (+ 157us) write_op.cc:270] Start()
0625 03:44:10.423245 (+ 24us) write_op.cc:276] Timestamp: P: 1750823032627607 usec, L: 0
0625 03:44:10.423415 (+ 170us) write_op.cc:270] Start()
0625 03:44:10.423438 (+ 23us) write_op.cc:276] Timestamp: P: 1750823032637929 usec, L: 0
0625 03:44:10.423653 (+ 215us) write_op.cc:270] Start()
0625 03:44:10.423675 (+ 22us) write_op.cc:276] Timestamp: P: 1750823032694803 usec, L: 0
0625 03:44:10.494094 (+ 70419us) write_op.cc:270] Start()
0625 03:44:10.494138 (+ 44us) write_op.cc:276] Timestamp: P: 1750823032739944 usec, L: 0
0625 03:44:10.494380 (+ 242us) write_op.cc:270] Start()
0625 03:44:10.494402 (+ 22us) write_op.cc:276] Timestamp: P: 1750823032772258 usec, L: 0
0625 03:44:10.494598 (+ 196us) write_op.cc:270] Start()
0625 03:44:10.494621 (+ 23us) write_op.cc:276] Timestamp: P: 1750823032815688 usec, L: 0
0625 03:44:10.494788 (+ 167us) write_op.cc:270] Start()
0625 03:44:10.494810 (+ 22us) write_op.cc:276] Timestamp: P: 1750823032831104 usec, L: 0
0625 03:44:10.494970 (+ 160us) write_op.cc:270] Start()
0625 03:44:10.494991 (+ 21us) write_op.cc:276] Timestamp: P: 1750823032888161 usec, L: 0
0625 03:44:10.495129 (+ 138us) write_op.cc:270] Start()
0625 03:44:10.495153 (+ 24us) write_op.cc:276] Timestamp: P: 1750823032947800 usec, L: 0
0625 03:44:10.495371 (+ 218us) write_op.cc:270] Start()
0625 03:44:10.495393 (+ 22us) write_op.cc:276] Timestamp: P: 1750823032966430 usec, L: 0
0625 03:44:10.495548 (+ 155us) write_op.cc:270] Start()
0625 03:44:10.495569 (+ 21us) write_op.cc:276] Timestamp: P: 1750823033009039 usec, L: 0
0625 03:44:10.509635 (+ 14066us) write_op.cc:270] Start()
0625 03:44:10.509676 (+ 41us) write_op.cc:276] Timestamp: P: 1750823033011505 usec, L: 0
0625 03:44:10.509947 (+ 271us) write_op.cc:270] Start()
0625 03:44:10.509973 (+ 26us) write_op.cc:276] Timestamp: P: 1750823033087417 usec, L: 0
0625 03:44:10.510211 (+ 238us) write_op.cc:270] Start()
0625 03:44:10.510233 (+ 22us) write_op.cc:276] Timestamp: P: 1750823033158771 usec, L: 0
0625 03:44:10.510396 (+ 163us) write_op.cc:270] Start()
0625 03:44:10.510416 (+ 20us) write_op.cc:276] Timestamp: P: 1750823033205137 usec, L: 0
0625 03:44:10.510604 (+ 188us) write_op.cc:270] Start()
0625 03:44:10.510632 (+ 28us) write_op.cc:276] Timestamp: P: 1750823033256909 usec, L: 0
0625 03:44:10.510773 (+ 141us) write_op.cc:270] Start()
0625 03:44:10.510792 (+ 19us) write_op.cc:276] Timestamp: P: 1750823033264673 usec, L: 0
0625 03:44:10.510953 (+ 161us) write_op.cc:270] Start()
0625 03:44:10.510973 (+ 20us) write_op.cc:276] Timestamp: P: 1750823033301349 usec, L: 0
0625 03:44:10.511149 (+ 176us) write_op.cc:270] Start()
0625 03:44:10.511169 (+ 20us) write_op.cc:276] Timestamp: P: 1750823033380835 usec, L: 0
0625 03:44:10.511352 (+ 183us) write_op.cc:270] Start()
0625 03:44:10.511373 (+ 21us) write_op.cc:276] Timestamp: P: 1750823033404355 usec, L: 0
0625 03:44:10.511529 (+ 156us) write_op.cc:270] Start()
0625 03:44:10.511549 (+ 20us) write_op.cc:276] Timestamp: P: 1750823033406397 usec, L: 0
0625 03:44:10.511828 (+ 279us) write_op.cc:270] Start()
0625 03:44:10.511852 (+ 24us) write_op.cc:276] Timestamp: P: 1750823033415305 usec, L: 0
0625 03:44:10.512010 (+ 158us) write_op.cc:270] Start()
0625 03:44:10.512036 (+ 26us) write_op.cc:276] Timestamp: P: 1750823033437040 usec, L: 0
0625 03:44:10.512220 (+ 184us) write_op.cc:270] Start()
0625 03:44:10.512247 (+ 27us) write_op.cc:276] Timestamp: P: 1750823033482034 usec, L: 0
0625 03:44:10.512438 (+ 191us) write_op.cc:270] Start()
0625 03:44:10.512458 (+ 20us) write_op.cc:276] Timestamp: P: 1750823033489395 usec, L: 0
0625 03:44:10.512639 (+ 181us) write_op.cc:270] Start()
0625 03:44:10.512660 (+ 21us) write_op.cc:276] Timestamp: P: 1750823033510627 usec, L: 0
0625 03:44:10.512820 (+ 160us) write_op.cc:270] Start()
0625 03:44:10.512839 (+ 19us) write_op.cc:276] Timestamp: P: 1750823033520374 usec, L: 0
0625 03:44:10.512975 (+ 136us) write_op.cc:270] Start()
0625 03:44:10.512994 (+ 19us) write_op.cc:276] Timestamp: P: 1750823033552761 usec, L: 0
0625 03:44:10.513128 (+ 134us) write_op.cc:270] Start()
0625 03:44:10.513148 (+ 20us) write_op.cc:276] Timestamp: P: 1750823033601052 usec, L: 0
0625 03:44:10.513370 (+ 222us) write_op.cc:270] Start()
0625 03:44:10.513392 (+ 22us) write_op.cc:276] Timestamp: P: 1750823033602502 usec, L: 0
0625 03:44:10.513542 (+ 150us) write_op.cc:270] Start()
0625 03:44:10.513561 (+ 19us) write_op.cc:276] Timestamp: P: 1750823033630461 usec, L: 0
0625 03:44:10.542604 (+ 29043us) write_op.cc:270] Start()
0625 03:44:10.542647 (+ 43us) write_op.cc:276] Timestamp: P: 1750823033637281 usec, L: 0
0625 03:44:10.542986 (+ 339us) write_op.cc:270] Start()
0625 03:44:10.543014 (+ 28us) write_op.cc:276] Timestamp: P: 1750823033668234 usec, L: 0
0625 03:44:10.543201 (+ 187us) write_op.cc:270] Start()
0625 03:44:10.543222 (+ 21us) write_op.cc:276] Timestamp: P: 1750823033678738 usec, L: 0
0625 03:44:10.543388 (+ 166us) write_op.cc:270] Start()
0625 03:44:10.543448 (+ 60us) write_op.cc:276] Timestamp: P: 1750823033707005 usec, L: 0
0625 03:44:10.543607 (+ 159us) write_op.cc:270] Start()
0625 03:44:10.543628 (+ 21us) write_op.cc:276] Timestamp: P: 1750823033737644 usec, L: 0
0625 03:44:10.543848 (+ 220us) write_op.cc:270] Start()
0625 03:44:10.543873 (+ 25us) write_op.cc:276] Timestamp: P: 1750823033767176 usec, L: 0
0625 03:44:10.544084 (+ 211us) write_op.cc:270] Start()
0625 03:44:10.544119 (+ 35us) write_op.cc:276] Timestamp: P: 1750823033787313 usec, L: 0
0625 03:44:10.544306 (+ 187us) write_op.cc:270] Start()
0625 03:44:10.544334 (+ 28us) write_op.cc:276] Timestamp: P: 1750823033797447 usec, L: 0
0625 03:44:10.544537 (+ 203us) write_op.cc:270] Start()
0625 03:44:10.544561 (+ 24us) write_op.cc:276] Timestamp: P: 1750823033851524 usec, L: 0
0625 03:44:10.544733 (+ 172us) write_op.cc:270] Start()
0625 03:44:10.544759 (+ 26us) write_op.cc:276] Timestamp: P: 1750823033874669 usec, L: 0
0625 03:44:10.544995 (+ 236us) write_op.cc:270] Start()
0625 03:44:10.545023 (+ 28us) write_op.cc:276] Timestamp: P: 1750823033881110 usec, L: 0
0625 03:44:10.545207 (+ 184us) write_op.cc:270] Start()
0625 03:44:10.545232 (+ 25us) write_op.cc:276] Timestamp: P: 1750823033915487 usec, L: 0
0625 03:44:10.545376 (+ 144us) write_op.cc:270] Start()
0625 03:44:10.545396 (+ 20us) write_op.cc:276] Timestamp: P: 1750823033931111 usec, L: 0
0625 03:44:10.545540 (+ 144us) write_op.cc:270] Start()
0625 03:44:10.545558 (+ 18us) write_op.cc:276] Timestamp: P: 1750823033939422 usec, L: 0
0625 03:44:10.545705 (+ 147us) write_op.cc:270] Start()
0625 03:44:10.545724 (+ 19us) write_op.cc:276] Timestamp: P: 1750823033965901 usec, L: 0
0625 03:44:10.545852 (+ 128us) write_op.cc:270] Start()
0625 03:44:10.545873 (+ 21us) write_op.cc:276] Timestamp: P: 1750823033973403 usec, L: 0
0625 03:44:10.546149 (+ 276us) write_op.cc:270] Start()
0625 03:44:10.546185 (+ 36us) write_op.cc:276] Timestamp: P: 1750823033997573 usec, L: 0
0625 03:44:10.546409 (+ 224us) write_op.cc:270] Start()
0625 03:44:10.546432 (+ 23us) write_op.cc:276] Timestamp: P: 1750823034027584 usec, L: 0
0625 03:44:10.546656 (+ 224us) write_op.cc:270] Start()
0625 03:44:10.546679 (+ 23us) write_op.cc:276] Timestamp: P: 1750823034029048 usec, L: 0
0625 03:44:10.546899 (+ 220us) write_op.cc:270] Start()
0625 03:44:10.546920 (+ 21us) write_op.cc:276] Timestamp: P: 1750823034051112 usec, L: 0
0625 03:44:10.547062 (+ 142us) write_op.cc:270] Start()
0625 03:44:10.547081 (+ 19us) write_op.cc:276] Timestamp: P: 1750823034108117 usec, L: 0
0625 03:44:10.557474 (+ 10393us) write_op.cc:270] Start()
0625 03:44:10.557508 (+ 34us) write_op.cc:276] Timestamp: P: 1750823034109387 usec, L: 0
0625 03:44:10.557739 (+ 231us) write_op.cc:270] Start()
0625 03:44:10.557759 (+ 20us) write_op.cc:276] Timestamp: P: 1750823034134868 usec, L: 0
0625 03:44:10.557919 (+ 160us) write_op.cc:270] Start()
0625 03:44:10.557938 (+ 19us) write_op.cc:276] Timestamp: P: 1750823034149889 usec, L: 0
0625 03:44:10.558389 (+ 451us) write_op.cc:270] Start()
0625 03:44:10.558417 (+ 28us) write_op.cc:276] Timestamp: P: 1750823034158284 usec, L: 0
0625 03:44:10.558575 (+ 158us) write_op.cc:270] Start()
0625 03:44:10.558594 (+ 19us) write_op.cc:276] Timestamp: P: 1750823034179247 usec, L: 0
0625 03:44:10.558710 (+ 116us) write_op.cc:270] Start()
0625 03:44:10.558725 (+ 15us) write_op.cc:276] Timestamp: P: 1750823034222743 usec, L: 0
0625 03:44:10.558894 (+ 169us) write_op.cc:270] Start()
0625 03:44:10.558913 (+ 19us) write_op.cc:276] Timestamp: P: 1750823034230079 usec, L: 0
0625 03:44:10.559067 (+ 154us) write_op.cc:270] Start()
0625 03:44:10.559086 (+ 19us) write_op.cc:276] Timestamp: P: 1750823034267373 usec, L: 0
0625 03:44:10.559259 (+ 173us) write_op.cc:270] Start()
0625 03:44:10.567658 (+ 8399us) write_op.cc:276] Timestamp: P: 1750823034284163 usec, L: 0
0625 03:44:10.571807 (+ 4149us) write_op.cc:270] Start()
0625 03:44:10.571844 (+ 37us) write_op.cc:276] Timestamp: P: 1750823034292937 usec, L: 0
0625 03:44:10.572119 (+ 275us) write_op.cc:270] Start()
0625 03:44:10.572145 (+ 26us) write_op.cc:276] Timestamp: P: 1750823034335411 usec, L: 0
0625 03:44:10.572400 (+ 255us) write_op.cc:270] Start()
0625 03:44:10.572425 (+ 25us) write_op.cc:276] Timestamp: P: 1750823034348485 usec, L: 0
0625 03:44:10.572611 (+ 186us) write_op.cc:270] Start()
0625 03:44:10.572636 (+ 25us) write_op.cc:276] Timestamp: P: 1750823034381933 usec, L: 0
0625 03:44:10.586371 (+ 13735us) write_op.cc:270] Start()
0625 03:44:10.586424 (+ 53us) write_op.cc:276] Timestamp: P: 1750823034383201 usec, L: 0
0625 03:44:10.586745 (+ 321us) write_op.cc:270] Start()
0625 03:44:10.586771 (+ 26us) write_op.cc:276] Timestamp: P: 1750823034416583 usec, L: 0
0625 03:44:10.587048 (+ 277us) write_op.cc:270] Start()
0625 03:44:10.587079 (+ 31us) write_op.cc:276] Timestamp: P: 1750823034438845 usec, L: 0
0625 03:44:10.587274 (+ 195us) write_op.cc:270] St
I20250625 03:44:11.576543 15266 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:44:11.589093 15266 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250625 03:44:17.381399 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 52638.9us
W20250625 03:44:17.812041 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.080s user 0.004s sys 0.000s
I20250625 03:44:18.389433 14733 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:44:18.409593 13678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:44:18.440948 14886 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:44:18.496053 13810 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:44:18.547915 14282 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 03:44:18.709419 14114 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
W20250625 03:44:19.392247 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.072s user 0.000s sys 0.000s
I20250625 03:44:19.680801 13870 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:39208 (request call id 5) took 1180 ms. Trace:
I20250625 03:44:19.681679 13870 rpcz_store.cc:276] 0625 03:44:18.500815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:44:18.500987 (+ 172us) service_pool.cc:225] Handling call
0625 03:44:19.680653 (+1179666us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250625 03:44:23.354323 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 44496.4us
W20250625 03:44:23.373557 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.060s user 0.001s sys 0.000s
W20250625 03:44:24.154893 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.098s user 0.001s sys 0.000s
W20250625 03:44:25.704416 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 66184.1us
W20250625 03:44:25.865340 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.004s sys 0.000s
W20250625 03:44:26.157852 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 45633.9us
W20250625 03:44:26.226996 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.055s user 0.003s sys 0.000s
W20250625 03:44:26.764336 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 72377.7us
I20250625 03:44:27.767302 13020 ts_manager.cc:284] Unset tserver state for 513d19d66cd54765940762233636fd31 from MAINTENANCE_MODE
I20250625 03:44:27.771200 13743 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:44:27.885650 13875 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:44:28.033067 13020 ts_manager.cc:284] Unset tserver state for d275425c3ef443f88abb12d24f4c2f2f from MAINTENANCE_MODE
I20250625 03:44:28.040499 14953 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:44:28.512658 14348 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:44:28.534696 14181 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:44:28.540500 14802 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:44:28.793730 13743 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:44:28.895064 13875 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
W20250625 03:44:29.172112 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.097s user 0.004s sys 0.000s
I20250625 03:44:30.846693 12990 maintenance_mode-itest.cc:745] Restarting batch of 2 tservers: c24f9318ff0c418292d80906d5f0fd95,506230f3ca974bf796583a6be750e38f
W20250625 03:44:33.693779 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 50978.7us
W20250625 03:44:34.201035 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 67219.3us
W20250625 03:44:35.531630 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 61991.9us
I20250625 03:44:36.138588 13022 ts_manager.cc:295] Set tserver state for c24f9318ff0c418292d80906d5f0fd95 to MAINTENANCE_MODE
I20250625 03:44:36.834259 13020 ts_manager.cc:295] Set tserver state for 506230f3ca974bf796583a6be750e38f to MAINTENANCE_MODE
W20250625 03:44:38.656152 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.072s user 0.004s sys 0.000s
W20250625 03:44:40.759789 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.003s sys 0.000s
I20250625 03:44:45.590852 13810 tablet_service.cc:1423] Tablet server c24f9318ff0c418292d80906d5f0fd95 set to quiescing
I20250625 03:44:45.592914 13810 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:44:45.974962 13678 tablet_service.cc:1423] Tablet server 506230f3ca974bf796583a6be750e38f set to quiescing
I20250625 03:44:45.979475 13678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 03:44:46.102725 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.121s user 0.000s sys 0.000s
W20250625 03:44:48.576645 13885 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.175.129:45233, user_credentials={real_user=slave}} blocked reactor thread for 366620us
W20250625 03:44:50.184713 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.079s user 0.001s sys 0.000s
W20250625 03:44:50.274320 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 52181.5us
I20250625 03:44:51.960482 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 13747
W20250625 03:44:51.970685 13885 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.175.129:45233, user_credentials={real_user=slave}} blocked reactor thread for 63723us
I20250625 03:44:52.058012 13869 generic_service.cc:196] Checking for leaks (request via RPC)
W20250625 03:44:52.964334 12990 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 13747: Timed out: CheckLeaks RPC to 127.12.175.134:40555 timed out after 1.000s (SENT)
I20250625 03:44:52.965018 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 13747
I20250625 03:44:53.017618 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.134:40555
--local_ip_for_outbound_sockets=127.12.175.134
--tserver_master_addrs=127.12.175.190:43735
--webserver_port=36347
--webserver_interface=127.12.175.134
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250625 03:44:53.872722 14232 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.133:36385, user_credentials={real_user=slave}} blocked reactor thread for 50787.8us
W20250625 03:44:54.697309 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.071s user 0.000s sys 0.000s
W20250625 03:44:55.091921 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 75578.1us
W20250625 03:44:55.160946 15242 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20250625 03:44:55.274942 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 177727us
W20250625 03:44:55.686082 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.058s user 0.004s sys 0.000s
I20250625 03:44:56.504737 15674 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:44:56.511914 15674 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:44:56.521823 13696 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 3746 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:44:56.536466 15674 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:44:56.538792 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 3746 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:44:56.559152 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 3746 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:44:56.560657 14134 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 3746 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:44:56.561793 14680 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:44:56.567073 15674 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250625 03:44:56.577792 15647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:44:56.578801 15647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:44:56.579247 15647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:44:56.595440 15647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:44:56.596029 15647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.134
I20250625 03:44:56.621515 15647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.134:40555
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.12.175.134
--webserver_port=36347
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.134
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:44:56.627612 15647 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:44:56.634402 15647 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:44:56.710009 15679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:44:56.712824 15681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:44:56.729350 15647 server_base.cc:1048] running on GCE node
W20250625 03:44:56.760231 15678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:44:56.808962 15647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:44:56.857636 15647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:44:56.870625 15647 hybrid_clock.cc:648] HybridClock initialized: now 1750823096866192 us; error 230 us; skew 500 ppm
I20250625 03:44:56.872033 15647 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:44:56.984293 15647 webserver.cc:469] Webserver started at http://127.12.175.134:36347/ using document root <none> and password file <none>
I20250625 03:44:57.001930 15647 fs_manager.cc:362] Metadata directory not provided
I20250625 03:44:57.002650 15647 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:44:57.281958 15647 fs_manager.cc:714] Time spent opening directory manager: real 0.211s user 0.011s sys 0.010s
I20250625 03:44:57.321460 15694 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 03:44:57.341557 15647 fs_manager.cc:730] Time spent opening block manager: real 0.053s user 0.007s sys 0.006s
I20250625 03:44:57.342100 15647 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal
uuid: "c24f9318ff0c418292d80906d5f0fd95"
format_stamp: "Formatted at 2025-06-25 03:43:01 on dist-test-slave-ttt3"
I20250625 03:44:57.348151 15647 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 03:44:57.488770 15647 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:44:57.491855 15647 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:44:57.492794 15647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:44:57.506636 15647 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:44:57.543242 15647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 03:44:57.543891 15647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.002s sys 0.000s
I20250625 03:44:57.544258 15647 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 03:44:57.544456 15647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 03:44:58.131084 13018 maintenance_manager.cc:419] P 7349fb30bd4546798ab374350c9b0bfa: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033394
I20250625 03:44:58.372846 13017 maintenance_manager.cc:643] P 7349fb30bd4546798ab374350c9b0bfa: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.240s user 0.057s sys 0.021s Metrics: {"bytes_written":7215,"cfile_init":1,"drs_written":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5068,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":11,"thread_start_us":15158,"threads_started":1,"wal-append.queue_time_us":14909}
I20250625 03:44:58.379809 13018 maintenance_manager.cc:419] P 7349fb30bd4546798ab374350c9b0bfa: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 723 bytes on disk
I20250625 03:44:58.383378 13017 maintenance_manager.cc:643] P 7349fb30bd4546798ab374350c9b0bfa: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20250625 03:45:00.573665 15647 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.134:40555
I20250625 03:45:00.573807 15825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.134:40555 every 8 connection(s)
I20250625 03:45:00.577787 15647 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data/info.pb
I20250625 03:45:00.591197 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 15647
I20250625 03:45:00.591769 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 13615
I20250625 03:45:00.707041 13738 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 03:45:00.721324 15826 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:45:00.722325 15826 heartbeater.cc:461] Registering TS with master...
I20250625 03:45:00.724545 15826 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:45:00.728546 13026 ts_manager.cc:194] Re-registered known tserver with Master: c24f9318ff0c418292d80906d5f0fd95 (127.12.175.134:40555)
I20250625 03:45:00.730093 13026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.134:47829
W20250625 03:45:01.594556 12990 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 13615: Timed out: CheckLeaks RPC to 127.12.175.133:36385 timed out after 1.000s (SENT)
I20250625 03:45:01.594851 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 13615
I20250625 03:45:01.744774 15826 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
W20250625 03:45:01.854616 14232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.175.133:36385 (error 108) [suppressed 23 similar messages]
I20250625 03:45:01.857272 12990 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
/tmp/dist-test-taskU8fiuI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.175.133:36385
--local_ip_for_outbound_sockets=127.12.175.133
--tserver_master_addrs=127.12.175.190:43735
--webserver_port=38511
--webserver_interface=127.12.175.133
--builtin_ntp_servers=127.12.175.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250625 03:45:01.886579 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.133:36385: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250625 03:45:02.296576 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.133:36385: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250625 03:45:02.845589 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.133:36385: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250625 03:45:03.326990 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.133:36385: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250625 03:45:03.408419 15840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:45:03.409119 15840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 03:45:03.409336 15840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:45:03.456696 15840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 03:45:03.457687 15840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.175.133
I20250625 03:45:03.507313 15840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.175.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.175.133:36385
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.12.175.133
--webserver_port=38511
--tserver_master_addrs=127.12.175.190:43735
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.175.133
--log_dir=/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 03:18:58 UTC on 5fd53c4cbb9d
build id 6756
ASAN enabled
I20250625 03:45:03.513592 15840 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 03:45:03.516916 15840 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 03:45:03.569832 15851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:45:03.573900 15853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 03:45:03.581352 15856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 03:45:03.584697 15840 server_base.cc:1048] running on GCE node
I20250625 03:45:03.587719 15840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 03:45:03.616313 15840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 03:45:03.655467 15840 hybrid_clock.cc:648] HybridClock initialized: now 1750823103655154 us; error 209 us; skew 500 ppm
I20250625 03:45:03.656441 15840 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 03:45:03.744567 15840 webserver.cc:469] Webserver started at http://127.12.175.133:38511/ using document root <none> and password file <none>
I20250625 03:45:03.746582 15840 fs_manager.cc:362] Metadata directory not provided
I20250625 03:45:03.746876 15840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 03:45:03.760342 15840 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.009s sys 0.000s
I20250625 03:45:03.822717 15862 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.002s user 0.000s sys 0.000s
I20250625 03:45:03.845940 15840 fs_manager.cc:730] Time spent opening block manager: real 0.083s user 0.014s sys 0.036s
I20250625 03:45:03.846510 15840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data,/tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal
uuid: "506230f3ca974bf796583a6be750e38f"
format_stamp: "Formatted at 2025-06-25 03:43:00 on dist-test-slave-ttt3"
I20250625 03:45:03.848160 15840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250625 03:45:03.891587 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.133:36385: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250625 03:45:04.001487 15840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 03:45:04.003614 15840 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 03:45:04.004196 15840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 03:45:04.069792 15840 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 03:45:04.089910 15869 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 03:45:04.102572 15840 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 03:45:04.102747 15840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.022s user 0.004s sys 0.004s
I20250625 03:45:04.102952 15840 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 03:45:04.109371 15840 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 03:45:04.109537 15840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.002s sys 0.003s
I20250625 03:45:04.110649 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap starting.
W20250625 03:45:04.464828 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.133:36385: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250625 03:45:05.018038 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.133:36385: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20250625 03:45:05.801359 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.133:36385: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20250625 03:45:06.598299 15840 rpc_server.cc:307] RPC server started. Bound to: 127.12.175.133:36385
I20250625 03:45:06.591637 15986 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.175.133:36385 every 8 connection(s)
W20250625 03:45:06.623144 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 94267.8us
I20250625 03:45:06.645414 15840 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data/info.pb
I20250625 03:45:06.652786 12990 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu as pid 15840
I20250625 03:45:06.787842 15988 heartbeater.cc:344] Connected to a master server at 127.12.175.190:43735
I20250625 03:45:06.789535 15988 heartbeater.cc:461] Registering TS with master...
I20250625 03:45:06.797963 15988 heartbeater.cc:507] Master 127.12.175.190:43735 requested a full tablet report, sending...
I20250625 03:45:06.836535 13026 ts_manager.cc:194] Re-registered known tserver with Master: 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385)
I20250625 03:45:06.853047 13026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.175.133:48021
I20250625 03:45:07.409178 16003 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:45:07.410848 16003 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:45:07.453337 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4217 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:45:07.459792 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4217 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:45:07.461251 16003 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:45:07.542934 15932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4217 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:45:07.611995 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:45:07.612643 14682 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 975eeb611762460bb2f09f5b4b71e3d4, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:45:07.613811 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4217 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:45:07.614663 16003 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:45:07.876030 15988 heartbeater.cc:499] Master 127.12.175.190:43735 was elected leader, sending a full tablet report...
W20250625 03:45:08.295614 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250625 03:45:09.596776 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250625 03:45:10.725878 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
I20250625 03:45:10.859395 14182 maintenance_manager.cc:419] P 975eeb611762460bb2f09f5b4b71e3d4: Scheduling FlushMRSOp(9126b7f95c9d434cb308581a7ce5b908): perf score=0.033362
W20250625 03:45:11.227846 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 56298.7us
W20250625 03:45:11.539534 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 49585.8us
I20250625 03:45:12.727972 15869 log.cc:826] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Log is configured to *not* fsync() on all Append() calls
I20250625 03:45:13.108918 16083 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 1 ops behind the committed index
I20250625 03:45:13.182017 16099 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:45:13.183468 16099 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:45:13.218361 16099 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:45:13.237426 14282 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 03:45:13.344424 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4342 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:45:13.345890 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4342 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:45:13.380168 14733 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:45:13.401834 14886 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:45:13.435508 15932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4342 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:45:13.434013 15913 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 03:45:13.439160 14063 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:45:13.439527 14063 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 975eeb611762460bb2f09f5b4b71e3d4; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:45:13.458492 16099 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:45:13.462713 14114 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250625 03:45:13.496150 14752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4342 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:45:13.606650 15748 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 03:45:13.680032 15869 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Append to log took a long time: real 0.229s user 0.004s sys 0.000s
W20250625 03:45:13.738011 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20250625 03:45:16.556939 16132 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 63 ops behind the committed index [suppressed 4 similar messages]
I20250625 03:45:16.729271 14349 maintenance_manager.cc:419] P 9cb1b4dfaea74515a9e5aab1fbc0870f: Scheduling FlushMRSOp(9126b7f95c9d434cb308581a7ce5b908): perf score=0.033383
W20250625 03:45:17.077010 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
I20250625 03:45:17.832651 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (117.2K/6.60M this segment, stats: ops{read=71 overwritten=0 applied=69 ignored=0} inserts{seen=3400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:45:18.463896 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250625 03:45:19.284458 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 43711us
W20250625 03:45:19.508558 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 71995.9us
W20250625 03:45:20.246919 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 44860.5us
I20250625 03:45:20.854786 16194 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 144 ops behind the committed index [suppressed 6 similar messages]
W20250625 03:45:20.929425 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 47830.8us
W20250625 03:45:20.945595 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 84217us
W20250625 03:45:22.477588 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.004s sys 0.000s
W20250625 03:45:22.542222 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
I20250625 03:45:22.822211 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (336.0K/6.60M this segment, stats: ops{read=201 overwritten=0 applied=199 ignored=0} inserts{seen=9900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250625 03:45:22.934057 14906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.12.175.129:45479 (request call id 2357) took 1538 ms. Trace:
I20250625 03:45:22.936116 16206 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:45:22.938755 16206 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:45:22.938450 14906 rpcz_store.cc:276] 0625 03:45:21.395665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:21.395758 (+ 93us) service_pool.cc:225] Handling call
0625 03:45:21.396218 (+ 460us) raft_consensus.cc:1469] Updating replica for 2 ops
0625 03:45:21.396492 (+ 274us) raft_consensus.cc:1532] Early marking committed up to index 4497
0625 03:45:21.410556 (+ 14064us) write_op.cc:489] Releasing partition, row and schema locks
0625 03:45:21.410563 (+ 7us) write_op.cc:454] Released schema lock
0625 03:45:21.410708 (+ 145us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0625 03:45:21.410920 (+ 212us) write_op.cc:270] Start()
0625 03:45:21.410960 (+ 40us) write_op.cc:276] Timestamp: P: 1750823121316497 usec, L: 0
0625 03:45:21.411255 (+ 295us) write_op.cc:270] Start()
0625 03:45:21.411294 (+ 39us) write_op.cc:276] Timestamp: P: 1750823121329051 usec, L: 0
0625 03:45:21.411666 (+ 372us) log.cc:844] Serialized 4934 byte log entry
0625 03:45:21.416255 (+ 4589us) raft_consensus.cc:1643] Marking committed up to 4499
0625 03:45:21.430734 (+ 14479us) write_op.cc:489] Releasing partition, row and schema locks
0625 03:45:21.430741 (+ 7us) write_op.cc:454] Released schema lock
0625 03:45:22.932459 (+1501718us) raft_consensus.cc:1695] Filling consensus response to leader.
0625 03:45:22.932879 (+ 420us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0625 03:45:22.933096 (+ 217us) raft_consensus.cc:1683] finished
0625 03:45:22.933117 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0625 03:45:22.933658 (+ 541us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 03:45:21.411790 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 9126b7f95c9d434cb308581a7ce5b908
0625 03:45:21.411980 (+ 190us) write_op.cc:432] Acquiring schema lock in shared mode
0625 03:45:21.411986 (+ 6us) write_op.cc:435] Acquired schema lock
0625 03:45:21.411989 (+ 3us) tablet.cc:660] Decoding operations
0625 03:45:21.412619 (+ 630us) write_op.cc:620] Acquiring the partition lock for write op
0625 03:45:21.412639 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0625 03:45:21.412645 (+ 6us) tablet.cc:683] Acquiring locks for 50 operations
0625 03:45:21.413007 (+ 362us) tablet.cc:699] Row locks acquired
0625 03:45:21.413010 (+ 3us) write_op.cc:260] PREPARE: finished
0625 03:45:21.425513 (+ 12503us) op_driver.cc:464] REPLICATION: finished
0625 03:45:21.425718 (+ 205us) write_op.cc:301] APPLY: starting
0625 03:45:21.425742 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0625 03:45:21.425966 (+ 224us) tablet.cc:1368] finished BulkCheckPresence
0625 03:45:21.425969 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 03:45:21.429840 (+ 3871us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 03:45:21.429973 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0625 03:45:21.429984 (+ 11us) write_op.cc:312] APPLY: finished
0625 03:45:21.430210 (+ 226us) log.cc:844] Serialized 320 byte log entry
0625 03:45:21.430321 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0625 03:45:21.430392 (+ 71us) write_op.cc:454] Released schema lock
0625 03:45:21.430406 (+ 14us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0625 03:45:21.413164 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 9126b7f95c9d434cb308581a7ce5b908
0625 03:45:21.413262 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0625 03:45:21.413267 (+ 5us) write_op.cc:435] Acquired schema lock
0625 03:45:21.413270 (+ 3us) tablet.cc:660] Decoding operations
0625 03:45:21.413912 (+ 642us) write_op.cc:620] Acquiring the partition lock for write op
0625 03:45:21.413926 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0625 03:45:21.413932 (+ 6us) tablet.cc:683] Acquiring locks for 50 operations
0625 03:45:21.414412 (+ 480us) tablet.cc:699] Row locks acquired
0625 03:45:21.414416 (+ 4us) write_op.cc:260] PREPARE: finished
0625 03:45:22.863115 (+1448699us) op_driver.cc:464] REPLICATION: finished
0625 03:45:22.898180 (+ 35065us) write_op.cc:301] APPLY: starting
0625 03:45:22.898230 (+ 50us) tablet.cc:1365] starting BulkCheckPresence
0625 03:45:22.898637 (+ 407us) tablet.cc:1368] finished BulkCheckPresence
0625 03:45:22.898641 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
Metrics: {"child_traces":[["op",{"apply.queue_time_us":64,"apply.run_cpu_time_us":4287,"apply.run_wall_time_us":4833,"num_ops":50,"partition_lock_wait_us":8,"prepare.queue_time_us":699,"prepare.run_cpu_time_us":1331,"prepare.run_wall_time_us":1330,"replication_time_us":5371}],["op",{"apply.queue_time_us":34721,"num_ops":50,"partition_lock_wait_us":4,"prepare.queue_time_us":1790,"prepare.run_cpu_time_us":1297,"prepare.run_wall_time_us":1368,"replication_time_us":1448191,"thread_start_us":68453,"threads_started":1}]]}
I20250625 03:45:23.007419 16206 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:45:23.166616 14752 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4499 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:45:23.177943 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4499 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:45:23.180598 14840 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:45:23.222630 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4499 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:45:23.225899 14841 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4
I20250625 03:45:23.227830 16206 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250625 03:45:23.258648 14838 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.175.129:45233 timed out after 0.314s (SENT)
W20250625 03:45:23.377461 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 61872.8us
I20250625 03:45:23.508525 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4499 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:45:23.512089 14301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4499 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
W20250625 03:45:23.588205 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 48480.1us
I20250625 03:45:23.883668 16212 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 192 ops behind the committed index [suppressed 4 similar messages]
W20250625 03:45:25.071292 14194 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4: Append to log took a long time: real 0.137s user 0.002s sys 0.000s
I20250625 03:45:25.420542 14092 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:38976 (request call id 28340) took 1418 ms. Trace:
I20250625 03:45:25.424979 14092 rpcz_store.cc:276] 0625 03:45:24.001912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:24.011102 (+ 9190us) service_pool.cc:225] Handling call
0625 03:45:24.026250 (+ 15148us) tablet_service.cc:3095] Found scanner 5f09e00f970844d997f088f03d11446a for tablet 9126b7f95c9d434cb308581a7ce5b908, query id is d33f82ec201f4fd79d5a7ed1195e8913
0625 03:45:25.420432 (+1394182us) inbound_call.cc:173] Queueing success response
Metrics: {"scanner_bytes_read":0}
W20250625 03:45:26.728643 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250625 03:45:26.884323 15869 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Append to log took a long time: real 0.060s user 0.004s sys 0.000s
W20250625 03:45:27.144169 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 51357.1us
W20250625 03:45:27.268401 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 61607.9us
I20250625 03:45:27.902467 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (506.3K/6.60M this segment, stats: ops{read=300 overwritten=0 applied=300 ignored=0} inserts{seen=14950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:45:28.380393 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.121s user 0.004s sys 0.000s
I20250625 03:45:28.468089 16202 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 256 ops behind the committed index [suppressed 4 similar messages]
W20250625 03:45:28.513970 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 73073.3us
W20250625 03:45:29.256469 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 43759.5us
W20250625 03:45:30.668741 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.057s user 0.004s sys 0.000s
W20250625 03:45:31.172315 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250625 03:45:32.156250 15869 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Append to log took a long time: real 0.064s user 0.000s sys 0.004s
I20250625 03:45:32.472026 16286 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 306 ops behind the committed index [suppressed 4 similar messages]
I20250625 03:45:32.571273 16295 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:45:32.572556 16295 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:45:32.580618 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4651 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:45:32.585733 14840 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:45:32.590984 16295 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:45:32.591116 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4651 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:45:32.598999 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4651 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:45:32.600629 14838 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:45:32.604075 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4651 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:45:32.836391 16295 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:45:33.065801 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (645.6K/6.60M this segment, stats: ops{read=382 overwritten=0 applied=380 ignored=0} inserts{seen=18950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:45:33.500324 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 54723.4us
W20250625 03:45:34.781431 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 64827.7us
W20250625 03:45:35.589546 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
I20250625 03:45:35.824517 16328 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 351 ops behind the committed index [suppressed 3 similar messages]
W20250625 03:45:35.889031 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 92623.8us
I20250625 03:45:36.578078 14733 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:45:36.619400 15913 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:45:36.672829 14114 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250625 03:45:36.714771 14282 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 03:45:36.829953 14886 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:45:36.867103 15748 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 03:45:36.993742 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 58318.8us
W20250625 03:45:37.743950 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.092s user 0.004s sys 0.000s
I20250625 03:45:37.925318 14796 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:44004 (request call id 5) took 1285 ms. Trace:
I20250625 03:45:37.925761 14796 rpcz_store.cc:276] 0625 03:45:36.640303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:36.640436 (+ 133us) service_pool.cc:225] Handling call
0625 03:45:37.925120 (+1284684us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 03:45:38.066834 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (782.1K/6.60M this segment, stats: ops{read=462 overwritten=0 applied=460 ignored=0} inserts{seen=22950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:45:38.719400 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 67899.5us
I20250625 03:45:39.347527 14343 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:37516 (request call id 5) took 2611 ms. Trace:
I20250625 03:45:39.438560 14343 rpcz_store.cc:276] 0625 03:45:36.736287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:36.736566 (+ 279us) service_pool.cc:225] Handling call
0625 03:45:39.347216 (+2610650us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 03:45:39.561115 16384 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 387 ops behind the committed index [suppressed 2 similar messages]
W20250625 03:45:39.595561 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 78847.8us
W20250625 03:45:39.711921 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.120s user 0.000s sys 0.004s
W20250625 03:45:41.591274 15869 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Append to log took a long time: real 0.059s user 0.004s sys 0.000s
W20250625 03:45:41.744541 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250625 03:45:42.201506 15869 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Append to log took a long time: real 0.057s user 0.004s sys 0.000s
I20250625 03:45:42.655157 16402 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 428 ops behind the committed index [suppressed 2 similar messages]
W20250625 03:45:42.682471 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 59978us
I20250625 03:45:43.077713 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (922.7K/6.60M this segment, stats: ops{read=545 overwritten=0 applied=544 ignored=0} inserts{seen=27150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250625 03:45:43.477854 16413 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:45:43.482982 16413 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:45:43.496551 16414 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:45:43.500797 16414 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:45:43.512172 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4780 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:45:43.516966 14840 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:45:43.520447 16413 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:45:43.523402 14134 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4780 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:45:43.524397 14134 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d275425c3ef443f88abb12d24f4c2f2f in term 4.
I20250625 03:45:43.526436 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4780 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:45:43.548462 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4780 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:45:43.551658 14838 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 975eeb611762460bb2f09f5b4b71e3d4, d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:45:43.555241 16413 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:45:43.570791 16415 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:45:43.584039 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4780 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:45:43.572302 16415 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:45:43.614326 16414 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:45:43.625648 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4780 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:45:43.628394 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4780 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:45:43.634236 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4780 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
W20250625 03:45:43.644790 14063 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:45:43.645480 14063 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 975eeb611762460bb2f09f5b4b71e3d4; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:45:43.658812 16414 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:45:43.687798 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4781 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:45:43.687906 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4781 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:45:43.686686 16415 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
W20250625 03:45:43.691002 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:45:43.693856 14905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4781 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:45:43.696663 14684 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:45:43.698264 14134 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4781 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:45:43.782146 16415 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250625 03:45:44.650380 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 75138.1us
W20250625 03:45:45.272096 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 83989.1us
W20250625 03:45:45.665966 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.066s user 0.002s sys 0.000s
W20250625 03:45:45.706429 15869 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Append to log took a long time: real 0.053s user 0.002s sys 0.000s
W20250625 03:45:45.822374 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20250625 03:45:45.938306 16429 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 467 ops behind the committed index [suppressed 3 similar messages]
W20250625 03:45:46.969347 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 46457.9us
I20250625 03:45:48.132226 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (1.10M/6.60M this segment, stats: ops{read=663 overwritten=0 applied=663 ignored=0} inserts{seen=33100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:45:48.319887 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 63975.7us
I20250625 03:45:49.172129 16430 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 506 ops behind the committed index [suppressed 2 similar messages]
W20250625 03:45:49.750804 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20250625 03:45:49.808475 14087 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:38976 (request call id 31354) took 1111 ms. Trace:
I20250625 03:45:49.809139 14087 rpcz_store.cc:276] 0625 03:45:48.696584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:48.696838 (+ 254us) service_pool.cc:225] Handling call
0625 03:45:48.696917 (+ 79us) tablet_service.cc:3095] Found scanner 5f09e00f970844d997f088f03d11446a for tablet 9126b7f95c9d434cb308581a7ce5b908, query id is d33f82ec201f4fd79d5a7ed1195e8913
0625 03:45:49.808247 (+1111330us) inbound_call.cc:173] Queueing success response
Metrics: {"scanner_bytes_read":0}
W20250625 03:45:51.499157 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 76658us
W20250625 03:45:51.757103 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 43855.4us
I20250625 03:45:51.789968 14093 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:38976 (request call id 31562) took 1228 ms. Trace:
I20250625 03:45:51.790921 14093 rpcz_store.cc:276] 0625 03:45:50.561734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:50.562919 (+ 1185us) service_pool.cc:225] Handling call
0625 03:45:50.563261 (+ 342us) tablet_service.cc:3095] Found scanner 89acfea76bef477e8e41776708af78f0 for tablet 9126b7f95c9d434cb308581a7ce5b908, query id is 270c805ecd384e13aa4c6a3d7ad43ef5
0625 03:45:51.789804 (+1226543us) inbound_call.cc:173] Queueing success response
Metrics: {"scanner_bytes_read":0}
W20250625 03:45:52.196218 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 54342.3us
I20250625 03:45:52.897603 16479 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 552 ops behind the committed index [suppressed 4 similar messages]
I20250625 03:45:53.159384 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (1.25M/6.60M this segment, stats: ops{read=756 overwritten=0 applied=756 ignored=0} inserts{seen=37750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:45:53.652878 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 96993.8us
I20250625 03:45:54.170948 14805 maintenance_manager.cc:419] P 513d19d66cd54765940762233636fd31: Scheduling FlushMRSOp(9126b7f95c9d434cb308581a7ce5b908): perf score=0.033363
W20250625 03:45:54.469702 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 56999.6us
W20250625 03:45:54.696518 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250625 03:45:54.754638 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 54645.1us
I20250625 03:45:55.203316 14087 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:38976 (request call id 31852) took 1364 ms. Trace:
I20250625 03:45:55.292981 14087 rpcz_store.cc:276] 0625 03:45:53.838796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:53.854811 (+ 16015us) service_pool.cc:225] Handling call
0625 03:45:53.855027 (+ 216us) tablet_service.cc:3095] Found scanner 61c665dbbff141f9a74255b413f659b9 for tablet 9126b7f95c9d434cb308581a7ce5b908, query id is aac623a2069f400aaa3e297adbaf67f0
0625 03:45:55.203087 (+1348060us) inbound_call.cc:173] Queueing success response
Metrics: {"scanner_bytes_read":0}
W20250625 03:45:55.612090 15869 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Append to log took a long time: real 0.073s user 0.000s sys 0.002s
I20250625 03:45:55.623270 16514 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:45:55.624603 16514 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:45:55.635923 15932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4924 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:45:55.639672 14840 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:45:55.677066 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4924 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:45:55.697643 16514 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:45:55.699627 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4924 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:45:55.704629 14134 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4924 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:45:55.710222 14839 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:45:55.724045 16514 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:45:56.066690 16495 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 581 ops behind the committed index [suppressed 3 similar messages]
I20250625 03:45:57.522837 14093 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:38976 (request call id 32004) took 1323 ms. Trace:
I20250625 03:45:57.523468 14093 rpcz_store.cc:276] 0625 03:45:56.199325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:56.206450 (+ 7125us) service_pool.cc:225] Handling call
0625 03:45:56.211979 (+ 5529us) tablet_service.cc:3095] Found scanner 61c665dbbff141f9a74255b413f659b9 for tablet 9126b7f95c9d434cb308581a7ce5b908, query id is aac623a2069f400aaa3e297adbaf67f0
0625 03:45:57.519240 (+1307261us) tablet_service.cc:3169] Deadline expired - responding early
0625 03:45:57.522721 (+ 3481us) inbound_call.cc:173] Queueing success response
Metrics: {"scanner_bytes_read":0}
I20250625 03:45:58.252420 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (1.31M/6.60M this segment, stats: ops{read=791 overwritten=0 applied=787 ignored=0} inserts{seen=39300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:45:58.304206 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 99509.7us
W20250625 03:45:58.582655 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.083s user 0.003s sys 0.000s
I20250625 03:45:58.672736 16556 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:45:58.674028 16556 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:45:58.686968 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4944 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:45:58.689031 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4944 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:45:58.691716 14840 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:45:58.701856 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4944 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:45:58.705832 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4944 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:45:58.726933 16556 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:45:58.729964 14839 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:45:58.746788 16556 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:45:59.163033 14282 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 03:45:59.222028 15748 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:45:59.276520 15913 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:45:59.349542 14114 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
W20250625 03:45:59.524554 15869 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Append to log took a long time: real 0.081s user 0.004s sys 0.000s
I20250625 03:45:59.558969 14886 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:45:59.686266 14733 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 03:46:00.559302 15821 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:49908 (request call id 5) took 1329 ms. Trace:
I20250625 03:46:00.562084 15821 rpcz_store.cc:276] 0625 03:45:59.228993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:59.229127 (+ 134us) service_pool.cc:225] Handling call
0625 03:46:00.558243 (+1329116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 03:46:00.775869 14948 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:39218 (request call id 5) took 1082 ms. Trace:
I20250625 03:46:00.776846 14948 rpcz_store.cc:276] 0625 03:45:59.644867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:59.645042 (+ 175us) service_pool.cc:225] Handling call
0625 03:46:00.727505 (+1082463us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250625 03:46:00.902016 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
I20250625 03:46:01.079844 16525 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 620 ops behind the committed index [suppressed 2 similar messages]
I20250625 03:46:01.616478 15977 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:43868 (request call id 6) took 1380 ms. Trace:
I20250625 03:46:01.628171 15977 rpcz_store.cc:276] 0625 03:46:00.236280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:46:00.249597 (+ 13317us) service_pool.cc:225] Handling call
0625 03:46:01.616286 (+1366689us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 03:46:01.695801 14176 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:53604 (request call id 5) took 2294 ms. Trace:
I20250625 03:46:01.705317 14176 rpcz_store.cc:276] 0625 03:45:59.400965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:59.401042 (+ 77us) service_pool.cc:225] Handling call
0625 03:46:01.695531 (+2294489us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250625 03:46:01.768144 13929 meta_cache.cc:1261] Time spent looking up entry by key: real 0.056s user 0.004s sys 0.000s
W20250625 03:46:02.013101 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 55150.2us
W20250625 03:46:02.647387 13928 meta_cache.cc:1261] Time spent looking up entry by key: real 0.099s user 0.001s sys 0.000s
I20250625 03:46:03.260656 14343 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:35956 (request call id 5) took 4036 ms. Trace:
I20250625 03:46:03.295974 14343 rpcz_store.cc:276] 0625 03:45:59.224571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:45:59.224752 (+ 181us) service_pool.cc:225] Handling call
0625 03:46:03.260508 (+4035756us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 03:46:03.356396 14176 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:53604 (request call id 6) took 1639 ms. Trace:
I20250625 03:46:03.357131 14176 rpcz_store.cc:276] 0625 03:46:01.712595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:46:01.713192 (+ 597us) service_pool.cc:225] Handling call
0625 03:46:03.351505 (+1638313us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 03:46:03.548604 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (1.38M/6.60M this segment, stats: ops{read=833 overwritten=0 applied=833 ignored=0} inserts{seen=41600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250625 03:46:03.952457 16598 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:03.961146 16598 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:04.071202 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4993 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:04.073680 15932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4993 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:46:04.078182 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:04.083627 16598 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:46:04.084244 14905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4993 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:46:04.089795 14684 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:46:04.096516 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4993 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:46:04.117147 16598 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:46:04.430974 16598 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:04.433557 16598 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:04.440439 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:04.441111 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:46:04.443856 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:04.448380 16598 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:46:04.448732 14905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:46:04.454021 14684 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:46:04.455101 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:46:04.455986 16598 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:46:04.521286 16606 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:04.523638 16606 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:04.526197 16608 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:04.527802 16608 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:04.562541 16587 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 650 ops behind the committed index [suppressed 3 similar messages]
I20250625 03:46:04.650529 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:46:04.655087 14063 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:04.622309 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:04.648594 14301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:04.708132 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:46:04.707388 14753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
W20250625 03:46:04.709878 14840 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:04.714402 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:46:04.716315 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:46:04.719087 14841 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:46:04.728080 14062 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 975eeb611762460bb2f09f5b4b71e3d4; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:46:04.736243 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 4995 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:46:04.733016 16608 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:46:04.750831 16616 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:46:04.764315 16606 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:46:04.772235 16608 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250625 03:46:05.337332 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.122s user 0.000s sys 0.000s
I20250625 03:46:05.384688 14343 rpcz_store.cc:275] Call kudu.server.GenericService.GetFlags from 127.0.0.1:35956 (request call id 6) took 2077 ms. Trace:
I20250625 03:46:05.392094 14343 rpcz_store.cc:276] 0625 03:46:03.306697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 03:46:03.306884 (+ 187us) service_pool.cc:225] Handling call
0625 03:46:05.384378 (+2077494us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250625 03:46:05.412330 13930 meta_cache.cc:1261] Time spent looking up entry by key: real 0.071s user 0.001s sys 0.000s
W20250625 03:46:06.760994 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/maintenance_mode-itest.cc:753: Failure
Failed
Bad status: Timed out: cluster didn't become healthy: Running 'cluster ksck 127.12.175.190:43735' did not succeed in 60 seconds: Runtime error: error running 'cluster ksck 127.12.175.190:43735': stdout: Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
7349fb30bd4546798ab374350c9b0bfa | 127.12.175.190:43735 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.175.148:41839 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server States
Server | State
----------------------------------+------------------
506230f3ca974bf796583a6be750e38f | MAINTENANCE_MODE
c24f9318ff0c418292d80906d5f0fd95 | MAINTENANCE_MODE
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
506230f3ca974bf796583a6be750e38f | 127.12.175.133:36385 | HEALTHY | /L0 | 0 | 0
513d19d66cd54765940762233636fd31 | 127.12.175.131:40037 | HEALTHY | /L0 | 0 | 0
975eeb611762460bb2f09f5b4b71e3d4 | 127.12.175.130:34057 | HEALTHY | /L0 | 0 | 3
9cb1b4dfaea74515a9e5aab1fbc0870f | 127.12.175.129:45233 | HEALTHY | /L0 | 1 | 0
c24f9318ff0c418292d80906d5f0fd95 | 127.12.175.134:40555 | HEALTHY | /L0 | 0 | 0
d275425c3ef443f88abb12d24f4c2f2f | 127.12.175.132:42473 | HEALTHY | /L0 | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/L0 | 6
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.12.175.129 | experimental | 127.12.175.129:45233
local_ip_for_outbound_sockets | 127.12.175.130 | experimental | 127.12.175.130:34057
local_ip_for_outbound_sockets | 127.12.175.131 | experimental | 127.12.175.131:40037
local_ip_for_outbound_sockets | 127.12.175.132 | experimental | 127.12.175.132:42473
local_ip_for_outbound_sockets | 127.12.175.133 | experimental | 127.12.175.133:36385
local_ip_for_outbound_sockets | 127.12.175.134 | experimental | 127.12.175.134:40555
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-0/data/info.pb | hidden | 127.12.175.129:45233
server_dump_info_path | /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-1/data/info.pb | hidden | 127.12.175.130:34057
server_dump_info_path | /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-2/data/info.pb | hidden | 127.12.175.131:40037
server_dump_info_path | /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-3/data/info.pb | hidden | 127.12.175.132:42473
server_dump_info_path | /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-4/data/info.pb | hidden | 127.12.175.133:36385
server_dump_info_path | /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0/minicluster-data/ts-5/data/info.pb | hidden | 127.12.175.134:40555
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.175.148:41839 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
Tablet 9126b7f95c9d434cb308581a7ce5b908 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): RUNNING [LEADER]
506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap replaying log segment 1/1 (1.31M/6.60M this segment, stats: ops{read=791 overwritten=0 applied=787 ignored=0} inserts{seen=39300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
513d19d66cd54765940762233636fd31 (127.12.175.131:40037): RUNNING
975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): RUNNING
d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): RUNNING
All reported replicas are:
A = 9cb1b4dfaea74515a9e5aab1fbc0870f
B = 506230f3ca974bf796583a6be750e38f
C = 513d19d66cd54765940762233636fd31
D = 975eeb611762460bb2f09f5b4b71e3d4
E = d275425c3ef443f88abb12d24f4c2f2f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------------------+--------------+--------------+------------
master | A* B C D E | | | Yes
A | A* B C D E | 4 | -1 | Yes
B | A B C D E | 4 | -1 | Yes
C | A* B C D E | 4 | -1 | Yes
D | A* B C D E | 4 | -1 | Yes
E | A* B C D E | 4 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 5 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Small | c24f9318ff0c418292d80906d5f0fd95 | 127.12.175.134:40555 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
, stderr: W20250625 03:45:54.584965 16439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 03:45:54.587370 16439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 03:45:54.857547 16439 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Runtime error: ksck discovered errors
: /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu: process exited with non-zero status 1
I20250625 03:46:07.025556 16640 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:07.030838 16640 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:07.038369 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5008 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:46:07.042032 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:07.047356 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5008 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:46:07.048367 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5008 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:07.050190 14905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5008 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:46:07.050853 16640 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:46:07.064352 14680 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:46:07.065515 16640 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:46:07.522318 14956 maintenance_manager.cc:419] P d275425c3ef443f88abb12d24f4c2f2f: Scheduling FlushMRSOp(9126b7f95c9d434cb308581a7ce5b908): perf score=0.033385
I20250625 03:46:07.577687 16609 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 1 similar messages]
I20250625 03:46:08.594889 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (1.57M/6.60M this segment, stats: ops{read=946 overwritten=0 applied=944 ignored=0} inserts{seen=47150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250625 03:46:10.725359 16634 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 2 similar messages]
W20250625 03:46:11.122188 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20250625 03:46:12.557099 16657 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:12.562706 16657 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:12.594552 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:46:12.602115 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:12.625531 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:46:12.631778 16657 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:46:12.634413 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:46:12.640384 14684 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 975eeb611762460bb2f09f5b4b71e3d4, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:46:12.660363 16657 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:46:13.052505 16657 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:13.053779 16657 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:13.061177 15932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:46:13.063540 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:13.065373 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:46:13.081408 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:46:13.085162 14684 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 975eeb611762460bb2f09f5b4b71e3d4, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:46:13.087714 16657 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:46:13.088533 16657 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:46:13.103060 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:13.109061 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
W20250625 03:46:13.171541 14684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 84073.2us
W20250625 03:46:13.551759 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
I20250625 03:46:13.607249 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (1.84M/6.60M this segment, stats: ops{read=1106 overwritten=0 applied=1105 ignored=0} inserts{seen=55200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250625 03:46:14.069684 16656 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 6 similar messages]
W20250625 03:46:15.399219 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 47392.6us
W20250625 03:46:15.442205 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 97839.8us
W20250625 03:46:16.388432 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250625 03:46:16.556885 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 51049us
I20250625 03:46:17.336247 16656 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 5 similar messages]
I20250625 03:46:18.625026 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (2.08M/6.60M this segment, stats: ops{read=1256 overwritten=0 applied=1256 ignored=0} inserts{seen=62700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:46:18.828850 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
I20250625 03:46:20.094394 16678 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:20.101040 16679 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:20.101927 16678 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:20.103853 16679 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:20.111991 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:20.111513 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:46:20.118341 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:20.119500 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:46:20.119948 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
W20250625 03:46:20.120532 14063 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:20.133118 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:46:20.133276 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:46:20.133740 14905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:46:20.135267 14684 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:46:20.136162 14062 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 975eeb611762460bb2f09f5b4b71e3d4; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:46:20.138258 16679 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:46:20.138733 16679 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:46:20.145592 16678 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:46:20.147622 16678 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:46:20.157418 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
W20250625 03:46:20.180222 14684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.12.175.132:42473, user_credentials={real_user=slave}} blocked reactor thread for 44957.1us
I20250625 03:46:20.585055 16668 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 5 similar messages]
W20250625 03:46:20.687443 14233 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.130:34057, user_credentials={real_user=slave}} blocked reactor thread for 48183.4us
W20250625 03:46:20.894040 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20250625 03:46:22.456518 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
I20250625 03:46:23.626415 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (2.43M/6.60M this segment, stats: ops{read=1462 overwritten=0 applied=1459 ignored=0} inserts{seen=72850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250625 03:46:23.806141 16685 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 9 similar messages]
W20250625 03:46:24.700273 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
W20250625 03:46:27.127799 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
I20250625 03:46:27.369159 16690 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 6 similar messages]
I20250625 03:46:28.716348 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (2.78M/6.60M this segment, stats: ops{read=1668 overwritten=0 applied=1668 ignored=0} inserts{seen=83300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:46:28.763008 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
W20250625 03:46:30.120306 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
I20250625 03:46:30.546337 16690 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 10 similar messages]
W20250625 03:46:31.850051 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
I20250625 03:46:33.604935 16690 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 7 similar messages]
I20250625 03:46:33.717160 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (3.05M/6.60M this segment, stats: ops{read=1828 overwritten=0 applied=1828 ignored=0} inserts{seen=91300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:46:33.738991 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
W20250625 03:46:35.680352 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
I20250625 03:46:36.872951 16699 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 8 similar messages]
W20250625 03:46:37.440477 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
I20250625 03:46:38.419435 12990 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250625 03:46:38.420406 12990 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 9cb1b4dfaea74515a9e5aab1fbc0870f and pid 14196
I20250625 03:46:38.720927 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (3.34M/6.60M this segment, stats: ops{read=1996 overwritten=0 applied=1994 ignored=0} inserts{seen=99600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:46:39.372511 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
I20250625 03:46:39.978969 16703 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 8 similar messages]
W20250625 03:46:40.584254 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
W20250625 03:46:41.781739 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
[New LWP 14214]
[New LWP 14215]
[New LWP 14216]
[New LWP 14217]
[New LWP 14223]
[New LWP 14224]
[New LWP 14225]
[New LWP 14228]
[New LWP 14229]
[New LWP 14232]
[New LWP 14233]
[New LWP 14234]
[New LWP 14235]
[New LWP 14237]
[New LWP 14238]
[New LWP 14239]
[New LWP 14240]
[New LWP 14241]
[New LWP 14242]
[New LWP 14243]
[New LWP 14244]
[New LWP 14245]
[New LWP 14248]
[New LWP 14249]
[New LWP 14250]
[New LWP 14251]
[New LWP 14252]
[New LWP 14253]
[New LWP 14254]
[New LWP 14255]
[New LWP 14256]
[New LWP 14257]
[New LWP 14258]
[New LWP 14259]
[New LWP 14260]
[New LWP 14261]
[New LWP 14262]
[New LWP 14263]
[New LWP 14265]
[New LWP 14266]
[New LWP 14267]
[New LWP 14268]
[New LWP 14269]
[New LWP 14271]
[New LWP 14272]
[New LWP 14273]
[New LWP 14274]
[New LWP 14275]
[New LWP 14276]
[New LWP 14277]
[New LWP 14278]
[New LWP 14279]
[New LWP 14280]
[New LWP 14281]
[New LWP 14282]
[New LWP 14283]
[New LWP 14284]
[New LWP 14285]
[New LWP 14286]
[New LWP 14287]
[New LWP 14288]
[New LWP 14289]
[New LWP 14290]
[New LWP 14291]
[New LWP 14292]
[New LWP 14293]
[New LWP 14294]
[New LWP 14295]
[New LWP 14296]
[New LWP 14297]
[New LWP 14298]
[New LWP 14299]
[New LWP 14300]
[New LWP 14301]
[New LWP 14302]
[New LWP 14303]
[New LWP 14304]
[New LWP 14305]
[New LWP 14306]
[New LWP 14307]
[New LWP 14308]
[New LWP 14309]
[New LWP 14310]
[New LWP 14311]
[New LWP 14312]
[New LWP 14313]
[New LWP 14314]
[New LWP 14315]
[New LWP 14316]
[New LWP 14317]
[New LWP 14318]
[New LWP 14319]
[New LWP 14320]
[New LWP 14321]
[New LWP 14322]
[New LWP 14323]
[New LWP 14324]
[New LWP 14325]
[New LWP 14326]
[New LWP 14327]
[New LWP 14328]
[New LWP 14329]
[New LWP 14331]
[New LWP 14332]
[New LWP 14333]
[New LWP 14334]
[New LWP 14335]
[New LWP 14336]
[New LWP 14337]
[New LWP 14338]
[New LWP 14339]
[New LWP 14340]
[New LWP 14341]
[New LWP 14342]
[New LWP 14343]
[New LWP 14344]
[New LWP 14345]
[New LWP 14346]
[New LWP 14347]
[New LWP 14348]
[New LWP 14349]
[New LWP 16703]
[New LWP 16705]
[New LWP 16710]
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
0x00007f1ad9faed50 in ?? ()
Id Target Id Frame
* 1 LWP 14196 "kudu" 0x00007f1ad9faed50 in ?? ()
2 LWP 14214 "kudu" 0x00007f1ad9faafb9 in ?? ()
3 LWP 14215 "kudu" 0x00007f1ad9faafb9 in ?? ()
4 LWP 14216 "kudu" 0x00007f1ad9faafb9 in ?? ()
5 LWP 14217 "kernel-watcher-" 0x00007f1ad9faafb9 in ?? ()
6 LWP 14223 "ntp client-1422" 0x00007f1ad9fae9e2 in ?? ()
7 LWP 14224 "file cache-evic" 0x00007f1ad9faafb9 in ?? ()
8 LWP 14225 "sq_acceptor" 0x00007f1ad2cd0cb9 in ?? ()
9 LWP 14228 "rpc reactor-142" 0x00007f1ad2cdda47 in ?? ()
10 LWP 14229 "rpc reactor-142" 0x00007f1ad2cdda47 in ?? ()
11 LWP 14232 "rpc reactor-142" 0x00007f1ad2cdda47 in ?? ()
12 LWP 14233 "rpc reactor-142" 0x00007f1ad2cdda47 in ?? ()
13 LWP 14234 "MaintenanceMgr " 0x00007f1ae3a00c57 in ?? ()
14 LWP 14235 "txn-status-mana" 0x00007f1ad9faafb9 in ?? ()
15 LWP 14237 "collect_and_rem" 0x00007f1ad9faafb9 in ?? ()
16 LWP 14238 "tc-session-exp-" 0x00007f1ad9faafb9 in ?? ()
17 LWP 14239 "rpc worker-1423" 0x00007f1ad9faaad3 in ?? ()
18 LWP 14240 "rpc worker-1424" 0x00007f1ad9faaad3 in ?? ()
19 LWP 14241 "rpc worker-1424" 0x00007f1ad9faaad3 in ?? ()
20 LWP 14242 "rpc worker-1424" 0x00007f1ad9faaad3 in ?? ()
21 LWP 14243 "rpc worker-1424" 0x00007f1ad9faaad3 in ?? ()
22 LWP 14244 "rpc worker-1424" 0x00007f1ad9faaad3 in ?? ()
23 LWP 14245 "rpc worker-1424" 0x00007f1ad9faaad3 in ?? ()
24 LWP 14248 "rpc worker-1424" 0x00007f1ad9faaad3 in ?? ()
25 LWP 14249 "rpc worker-1424" 0x00007f1ad9faaad3 in ?? ()
26 LWP 14250 "rpc worker-1425" 0x00007f1ad9faaad3 in ?? ()
27 LWP 14251 "rpc worker-1425" 0x00007f1ad9faaad3 in ?? ()
28 LWP 14252 "rpc worker-1425" 0x00007f1ad9faaad3 in ?? ()
29 LWP 14253 "rpc worker-1425" 0x00007f1ad9faaad3 in ?? ()
30 LWP 14254 "rpc worker-1425" 0x00007f1ad9faaad3 in ?? ()
31 LWP 14255 "rpc worker-1425" 0x00007f1ad9faaad3 in ?? ()
32 LWP 14256 "rpc worker-1425" 0x00007f1ad9faaad3 in ?? ()
33 LWP 14257 "rpc worker-1425" 0x00007f1ad9faaad3 in ?? ()
34 LWP 14258 "rpc worker-1425" 0x00007f1ad9faaad3 in ?? ()
35 LWP 14259 "rpc worker-1425" 0x00007f1ad9faaad3 in ?? ()
36 LWP 14260 "rpc worker-1426" 0x00007f1ad9faaad3 in ?? ()
37 LWP 14261 "rpc worker-1426" 0x00007f1ad9faaad3 in ?? ()
38 LWP 14262 "rpc worker-1426" 0x00007f1ad9faaad3 in ?? ()
39 LWP 14263 "rpc worker-1426" 0x00007f1ad9faaad3 in ?? ()
40 LWP 14265 "rpc worker-1426" 0x00007f1ad9faaad3 in ?? ()
41 LWP 14266 "rpc worker-1426" 0x00007f1ad9faaad3 in ?? ()
42 LWP 14267 "rpc worker-1426" 0x00007f1ad9faaad3 in ?? ()
43 LWP 14268 "rpc worker-1426" 0x00007f1ad9faaad3 in ?? ()
44 LWP 14269 "rpc worker-1426" 0x00007f1ad9faaad3 in ?? ()
45 LWP 14271 "rpc worker-1427" 0x00007f1ad9faaad3 in ?? ()
46 LWP 14272 "rpc worker-1427" 0x00007f1ad9faaad3 in ?? ()
47 LWP 14273 "rpc worker-1427" 0x00007f1ad9faaad3 in ?? ()
48 LWP 14274 "rpc worker-1427" 0x00007f1ad9faaad3 in ?? ()
49 LWP 14275 "rpc worker-1427" 0x00007f1ad9faaad3 in ?? ()
50 LWP 14276 "rpc worker-1427" 0x00007f1ad9faaad3 in ?? ()
51 LWP 14277 "rpc worker-1427" 0x00007f1ad9faaad3 in ?? ()
52 LWP 14278 "rpc worker-1427" 0x00007f1ad9faaad3 in ?? ()
53 LWP 14279 "rpc worker-1427" 0x00007f1ad9faaad3 in ?? ()
54 LWP 14280 "rpc worker-1428" 0x00007f1ad9faaad3 in ?? ()
55 LWP 14281 "rpc worker-1428" 0x00007f1ad9faaad3 in ?? ()
56 LWP 14282 "rpc worker-1428" 0x00007f1ad9faaad3 in ?? ()
57 LWP 14283 "rpc worker-1428" 0x00007f1ad9faaad3 in ?? ()
58 LWP 14284 "rpc worker-1428" 0x00007f1ad9faaad3 in ?? ()
59 LWP 14285 "rpc worker-1428" 0x00007f1ad9faaad3 in ?? ()
60 LWP 14286 "rpc worker-1428" 0x00007f1ad9faaad3 in ?? ()
61 LWP 14287 "rpc worker-1428" 0x00007f1ad9faaad3 in ?? ()
62 LWP 14288 "rpc worker-1428" 0x00007f1ad9faaad3 in ?? ()
63 LWP 14289 "rpc worker-1428" 0x00007f1ad9faaad3 in ?? ()
64 LWP 14290 "rpc worker-1429" 0x00007f1ad9faaad3 in ?? ()
65 LWP 14291 "rpc worker-1429" 0x00007f1ad9faaad3 in ?? ()
66 LWP 14292 "rpc worker-1429" 0x00007f1ad9faaad3 in ?? ()
67 LWP 14293 "rpc worker-1429" 0x00007f1ad9faaad3 in ?? ()
68 LWP 14294 "rpc worker-1429" 0x00007f1ad9faaad3 in ?? ()
69 LWP 14295 "rpc worker-1429" 0x00007f1ad9faaad3 in ?? ()
70 LWP 14296 "rpc worker-1429" 0x00007f1ad9faaad3 in ?? ()
71 LWP 14297 "rpc worker-1429" 0x00007f1ad9faaad3 in ?? ()
72 LWP 14298 "rpc worker-1429" 0x00007f1ad9faaad3 in ?? ()
73 LWP 14299 "rpc worker-1429" 0x00007f1ad9faaad3 in ?? ()
74 LWP 14300 "rpc worker-1430" 0x00007f1ad9faaad3 in ?? ()
75 LWP 14301 "rpc worker-1430" 0x00007f1ad9faaad3 in ?? ()
76 LWP 14302 "rpc worker-1430" 0x00007f1ad9faaad3 in ?? ()
77 LWP 14303 "rpc worker-1430" 0x00007f1ad9faaad3 in ?? ()
78 LWP 14304 "rpc worker-1430" 0x00007f1ad9faaad3 in ?? ()
79 LWP 14305 "rpc worker-1430" 0x00007f1ad9faaad3 in ?? ()
80 LWP 14306 "rpc worker-1430" 0x00007f1ad9faaad3 in ?? ()
81 LWP 14307 "rpc worker-1430" 0x00007f1ad9faaad3 in ?? ()
82 LWP 14308 "rpc worker-1430" 0x00007f1ad9faaad3 in ?? ()
83 LWP 14309 "rpc worker-1430" 0x00007f1ad9faaad3 in ?? ()
84 LWP 14310 "rpc worker-1431" 0x00007f1ad9faaad3 in ?? ()
85 LWP 14311 "rpc worker-1431" 0x00007f1ad9faaad3 in ?? ()
86 LWP 14312 "rpc worker-1431" 0x00007f1ad9faaad3 in ?? ()
87 LWP 14313 "rpc worker-1431" 0x00007f1ad9faaad3 in ?? ()
88 LWP 14314 "rpc worker-1431" 0x00007f1ad9faaad3 in ?? ()
89 LWP 14315 "rpc worker-1431" 0x00007f1ad9faaad3 in ?? ()
90 LWP 14316 "rpc worker-1431" 0x00007f1ad9faaad3 in ?? ()
91 LWP 14317 "rpc worker-1431" 0x00007f1ad9faaad3 in ?? ()
92 LWP 14318 "rpc worker-1431" 0x00007f1ad9faaad3 in ?? ()
93 LWP 14319 "rpc worker-1431" 0x00007f1ad9faaad3 in ?? ()
94 LWP 14320 "rpc worker-1432" 0x00007f1ad9faaad3 in ?? ()
95 LWP 14321 "rpc worker-1432" 0x00007f1ad9faaad3 in ?? ()
96 LWP 14322 "rpc worker-1432" 0x00007f1ad9faaad3 in ?? ()
97 LWP 14323 "rpc worker-1432" 0x00007f1ad9faaad3 in ?? ()
98 LWP 14324 "rpc worker-1432" 0x00007f1ad9faaad3 in ?? ()
99 LWP 14325 "rpc worker-1432" 0x00007f1ad9faaad3 in ?? ()
100 LWP 14326 "rpc worker-1432" 0x00007f1ad9faaad3 in ?? ()
101 LWP 14327 "rpc worker-1432" 0x00007f1ad9faaad3 in ?? ()
102 LWP 14328 "rpc worker-1432" 0x00007f1ad9faaad3 in ?? ()
103 LWP 14329 "rpc worker-1432" 0x00007f1ad9faaad3 in ?? ()
104 LWP 14331 "rpc worker-1433" 0x00007f1ad9faaad3 in ?? ()
105 LWP 14332 "rpc worker-1433" 0x00007f1ad9faaad3 in ?? ()
106 LWP 14333 "rpc worker-1433" 0x00007f1ad9faaad3 in ?? ()
107 LWP 14334 "rpc worker-1433" 0x00007f1ad9faaad3 in ?? ()
108 LWP 14335 "rpc worker-1433" 0x00007f1ad9faaad3 in ?? ()
109 LWP 14336 "rpc worker-1433" 0x00007f1ad9faaad3 in ?? ()
110 LWP 14337 "rpc worker-1433" 0x00007f1ad9faaad3 in ?? ()
111 LWP 14338 "rpc worker-1433" 0x00007f1ad9faaad3 in ?? ()
112 LWP 14339 "rpc worker-1433" 0x00007f1ad9faaad3 in ?? ()
113 LWP 14340 "rpc worker-1434" 0x00007f1ad9faaad3 in ?? ()
114 LWP 14341 "rpc worker-1434" 0x00007f1ad9faaad3 in ?? ()
115 LWP 14342 "rpc worker-1434" 0x00007f1ad9faaad3 in ?? ()
116 LWP 14343 "rpc worker-1434" 0x00007f1ad9faaad3 in ?? ()
117 LWP 14344 "diag-logger-143" 0x00007f1ad9faafb9 in ?? ()
118 LWP 14345 "result-tracker-" 0x00007f1ad9faafb9 in ?? ()
119 LWP 14346 "excess-log-dele" 0x00007f1ad9faafb9 in ?? ()
120 LWP 14347 "acceptor-14347" 0x00007f1ad2cdf0c7 in ?? ()
121 LWP 14348 "heartbeat-14348" 0x00007f1ad9faafb9 in ?? ()
122 LWP 14349 "maintenance_sch" 0x00007f1ad9faafb9 in ?? ()
123 LWP 16703 "raft [worker]-1" 0x00007f1ad9faafb9 in ?? ()
124 LWP 16705 "raft [worker]-1" 0x00007f1ad9faafb9 in ?? ()
125 LWP 16710 "raft [worker]-1" 0x00007f1ad9faafb9 in ?? ()
Thread 125 (LWP 16710):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000017 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007f1a867b1ba4 in ?? ()
#5 0x00007f1a867b1830 in ?? ()
#6 0x000000000000002f in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f1a867b1850 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 124 (LWP 16705):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000062 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007f1a85fb0ba0 in ?? ()
#5 0x00007f1a85fb0830 in ?? ()
#6 0x00000000000000c4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 123 (LWP 16703):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000004d in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007f1a7c6a2ba4 in ?? ()
#5 0x00007f1a7c6a2830 in ?? ()
#6 0x000000000000009b in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f1a7c6a2850 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 122 (LWP 14349):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x00007f1a8eed86e0 in ?? ()
#2 0x0000000000000340 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f1a8eed8730 in ?? ()
#6 0x0000000000000680 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 14348):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x8272756aa36a08ac in ?? ()
#2 0x00000000000000d2 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007f1a8f6f05f0 in ?? ()
#6 0x00000000000001a4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 14347):
#0 0x00007f1ad2cdf0c7 in ?? ()
#1 0x00006140000212a8 in ?? ()
#2 0x00007f1a8ff17d90 in ?? ()
#3 0x00007f1a8ff17dc0 in ?? ()
#4 0x00007f1a8ff17ec0 in ?? ()
#5 0x00007f1a8ff17db0 in ?? ()
#6 0x00007f1a8ff17e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b082b in __sanitizer::theDepot ()
#9 0x00007f1ad6b653a0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 14346):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x00007f1a90730f40 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffc62aee7f0 in ?? ()
#5 0x00007f1a90730f30 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 14345):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x00007f1a90f49100 in ?? ()
#2 0x00000000000000d1 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f1a90f490f0 in ?? ()
#6 0x00000000000001a2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 14344):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 14343):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002ffff8 in ?? ()
#4 0x00007f1a91f79e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1a91f79eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 14342):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002f6808 in ?? ()
#4 0x00007f1a92791e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1a92791eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 14341):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 14340):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 14339):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 14338):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 14337):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 14336):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 14335):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 14334):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 14333):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 14332):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 14331):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 14329):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 14328):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 14327):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 14326):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 14325):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 14324):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 14323):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 14322):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 14321):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 14320):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 14319):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 14318):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 14317):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 14316):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 14315):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 14314):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 14313):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 14312):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 14311):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 14310):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 14309):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 14308):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 14307):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 14306):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 14305):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 14304):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 14303):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 14302):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x00000000000000fc in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001b6878 in ?? ()
#4 0x00007f1aa5b22e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1aa5b22eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 14301):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000116 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001b0078 in ?? ()
#4 0x00007f1aa633ae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1aa633aeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 14300):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000081 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f1aa6b52e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1aa6b52eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f1ad9faa770 in ?? ()
#10 0x00007f1aa6b52eb0 in ?? ()
#11 0x00000fe354d6a5d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 73 (LWP 14299):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 14298):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 14297):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 14296):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 14295):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 14294):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 14293):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 14292):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 14291):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 14290):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 14289):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 14288):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 14287):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 14286):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 14285):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 14284):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 14283):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 14282):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000130028 in ?? ()
#4 0x00007f1aafd13e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1aafd13eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 14281):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 14280):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 14279):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 14278):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 14277):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 14276):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 14275):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 14274):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 14273):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 14272):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 14271):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 14269):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 14268):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 14267):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 14266):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 14265):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 14263):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 14262):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 14261):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 14260):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x00000000000005c5 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f1aba82ee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1aba82eeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f1ad9faa770 in ?? ()
#10 0x00007f1aba82eeb0 in ?? ()
#11 0x00000fe357505dd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 14259):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000451 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007f1abb046e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1abb046eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007f1ad9faa770 in ?? ()
#10 0x00007f1abb046eb0 in ?? ()
#11 0x00000fe357608dd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 14258):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x00000000000004db in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000967fc in ?? ()
#4 0x00007f1abb85ee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1abb85eeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000967b0 in ?? ()
#9 0x00007f1ad9faa770 in ?? ()
#10 0x00007f1abb85eeb0 in ?? ()
#11 0x00000fe35770bdd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 33 (LWP 14257):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 14256):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 14255):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 14254):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 14253):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 14252):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 14251):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 14250):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 14249):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 14248):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 14245):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 14244):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 14243):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 14242):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 14241):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 14240):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 14239):
#0 0x00007f1ad9faaad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 14238):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x00007f1ac4218cc0 in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a560 in ?? ()
#5 0x00007f1ac4218cb0 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 14237):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x0000000000000000 in ?? ()
I20250625 03:46:43.373822 16715 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
Thread 14 (LWP 14235):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x00007f1ac5a9f240 in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d1a8 in ?? ()
#5 0x00007f1ac5a9f230 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 14234):
#0 0x00007f1ae3a00c57 in ?? ()
#1 0x0000000041b58ab3 in ?? ()
#2 0x00007f1ae3313d4a in ?? ()
#3 0x00007f1ae3a009e0 in ?? ()
#4 0x00007f1ac62ae0a0 in ?? ()
#5 0x0000000000000003 in ?? ()
#6 0x00007f1ac62ad050 in ?? ()
#7 0x000061400019a240 in ?? ()
#8 0x000062d002986520 in ?? ()
#9 0x0000000000000003 in ?? ()
#10 0x00007f1ae3a0638d in ?? ()
I20250625 03:46:43.376139 16715 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
#11 0x000060800011faa0 in ?? ()
#12 0x00007f1ac62ac1f0 in ?? ()
#13 0x00007f1ac62ad040 in ?? ()
#14 0x00007f1ac62ad160 in ?? ()
#15 0x00007f1ac62ad010 in ?? ()
#16 0x000000000095b82e in __sanitizer::theDepot ()
#17 0x00000fe35a80a700 in ?? ()
#18 0x00007f1ac62ad160 in ?? ()
I20250625 03:46:43.383070 16714 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
#19 0x00007f1ac62ad1f0 in ?? ()
#20 0x00007f1ae39b169b in ?? ()
#21 0x0000000045e0360e in ?? ()
#22 0x00007f1ae32e5816 in ?? ()
#23 0x00007f1ae39b12f0 in ?? ()
#24 0x00007f1ac62ac6c0 in ?? ()
#25 0x00007f1a00029cf7 in ?? ()
#26 0x00007f1ac62ac140 in ?? ()
#27 0x0000000000000000 in ?? ()
I20250625 03:46:43.384426 16714 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
Thread 12 (LWP 14233):
#0 0x00007f1ad2cdda47 in ?? ()
#1 0x00007f1ac6ab2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f1ac6ab2310 in ?? ()
#4 0x00007f1ac6ab2520 in ?? ()
#5 0x00007f1ac6ab2360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f1ac6ab23e0 in ?? ()
#8 0x00007f1ad535025d in ?? ()
#9 0x3f7dd8c400c80000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x000000006a000000 in ?? ()
I20250625 03:46:43.388876 16715 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
#14 0x00007f1a00000000 in ?? ()
#15 0x41da16db2f4cc7ae in ?? ()
#16 0x00000fe3d8d4e47c in ?? ()
#17 0x00007f1ac6ab23c0 in ?? ()
#18 0x00007f1ad5354ba3 in ?? ()
#19 0x00007f1ac6ab2390 in ?? ()
#20 0x3f7dd8c400c80000 in ?? ()
#21 0x00000000c6ab23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3f7dd8c400c80000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 14232):
#0 0x00007f1ad2cdda47 in ?? ()
#1 0x00007f1ac72b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f1ac72b3310 in ?? ()
I20250625 03:46:43.392833 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
#4 0x00007f1ac72b3520 in ?? ()
#5 0x00007f1ac72b3360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f1ac72b33e0 in ?? ()
I20250625 03:46:43.393748 14132 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 4.
#8 0x00007f1ad535025d in ?? ()
#9 0x3fa2f9cbb2dc8000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x0000000023f00000 in ?? ()
#14 0x00007f1a00000000 in ?? ()
#15 0x41da16db2f4cc7ad in ?? ()
#16 0x00000fe3d8e4e67c in ?? ()
#17 0x00007f1ac72b33c0 in ?? ()
#18 0x00007f1ad5354ba3 in ?? ()
#19 0x00007f1ac72b3390 in ?? ()
#20 0x3fa2f9cbb2dc8000 in ?? ()
#21 0x00000000c72b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fa2f9cbb2dc8000 in ?? ()
I20250625 03:46:43.397852 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 14229):
#0 0x00007f1ad2cdda47 in ?? ()
#1 0x00007f1ac7ab4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f1ac7ab4310 in ?? ()
#4 0x00007f1ac7ab4520 in ?? ()
#5 0x00007f1ac7ab4360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f1ac7ab43e0 in ?? ()
#8 0x00007f1ad535025d in ?? ()
#9 0x3f923cdd56070000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x000000001e000000 in ?? ()
I20250625 03:46:43.400919 16714 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
#14 0x00007f1a00000000 in ?? ()
#15 0x41da16db2f4cc7ad in ?? ()
#16 0x00000fe3d8f4e87c in ?? ()
#17 0x00007f1ac7ab43c0 in ?? ()
#18 0x00007f1ad5354ba3 in ?? ()
#19 0x00007f1ac7ab4390 in ?? ()
#20 0x3f923cdd56070000 in ?? ()
#21 0x00000000c7ab43e0 in ?? ()
I20250625 03:46:43.400832 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021098 in ?? ()
#24 0x3f923cdd56070000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 14228):
#0 0x00007f1ad2cdda47 in ?? ()
#1 0x00007f1ac9aad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f1ac9aad310 in ?? ()
#4 0x00007f1ac9aad520 in ?? ()
#5 0x00007f1ac9aad360 in ?? ()
#6 0x0000614000020e98 in ?? ()
#7 0x00007f1ac9aad3e0 in ?? ()
#8 0x00007f1ad535025d in ?? ()
#9 0x3fb9725ad4dcc000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x000000009e000000 in ?? ()
#14 0x00007f1a00000000 in ?? ()
#15 0x41da16db2f4cc7b0 in ?? ()
#16 0x00000fe3d934da7c in ?? ()
#17 0x00007f1ac9aad3c0 in ?? ()
#18 0x00007f1ad5354ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
W20250625 03:46:43.402968 14063 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
Thread 8 (LWP 14225):
#0 0x00007f1ad2cd0cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f1acb4b77c8 in ?? ()
#3 0x000060200001ca10 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a8471 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 14224):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x00007f1acacb60c0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007f1acacb60b0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 14223):
#0 0x00007f1ad9fae9e2 in ?? ()
#1 0x00007f1aca4b3ba0 in ?? ()
#2 0x00007f1aca4b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f1aca4b3bf0 in ?? ()
#6 0x000000000177e760 in ?? ()
#7 0x00007f1aca4b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f1aca4b3c40 in ?? ()
#10 0x00000000008a2967 in __sanitizer::theDepot ()
#11 0x00007f1adf8c1dcc in ?? ()
#12 0x00007f1adf8b17c9 in ?? ()
#13 0x00007f1adf8b5db6 in ?? ()
#14 0x00007f1adf8ba7f0 in ?? ()
#15 0x00007f1adf8ba619 in ?? ()
#16 0x0000000000a751bd in __sanitizer::theDepot ()
#17 0x00007f1ad6b85f1e in ?? ()
#18 0x00007f1ad9fa46db in ?? ()
#19 0x00000fe3d948e684 in ?? ()
#20 0x00007f1aca4b3440 in ?? ()
I20250625 03:46:43.405627 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 14217):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x00007f1acc4b8c80 in ?? ()
#2 0x0000000000000414 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007f1acc4b8c70 in ?? ()
I20250625 03:46:43.406404 14751 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975eeb611762460bb2f09f5b4b71e3d4 in term 4.
#6 0x0000000000000828 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 14216):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 14215):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 14214):
#0 0x00007f1ad9faafb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f1acdcbc450 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 14196):
#0 0x00007f1ad9faed50 in ?? ()
#1 0x0000000000000000 in ?? ()
I20250625 03:46:43.409780 14905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:46:43.412303 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:46:43.414155 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:43.431555 14300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:43.434254 14301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:43.437041 14061 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:46:43.437644 14680 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:46:43.439174 16715 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:46:43.440274 16714 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250625 03:46:43.469439 16716 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:43.480814 16716 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:43.499366 14301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:43.503970 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:46:43.505084 14840 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:43.512899 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:46:43.516683 14839 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:46:43.510857 16716 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:46:43.519357 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:46:43.519937 16710 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 11 similar messages]
I20250625 03:46:43.519454 16716 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
************************* END STACKS ***************************
I20250625 03:46:43.612987 12990 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID 975eeb611762460bb2f09f5b4b71e3d4 and pid 14038
W20250625 03:46:43.704800 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
I20250625 03:46:44.035324 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (3.59M/6.60M this segment, stats: ops{read=2146 overwritten=0 applied=2146 ignored=0} inserts{seen=107200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:46:44.884317 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
W20250625 03:46:46.692778 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
I20250625 03:46:46.814543 16724 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 10 similar messages]
W20250625 03:46:48.071313 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
I20250625 03:46:49.065837 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (3.81M/6.60M this segment, stats: ops{read=2273 overwritten=0 applied=2273 ignored=0} inserts{seen=113550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
[New LWP 14045]
[New LWP 14046]
[New LWP 14047]
[New LWP 14048]
[New LWP 14054]
[New LWP 14055]
[New LWP 14056]
[New LWP 14061]
[New LWP 14062]
[New LWP 14063]
[New LWP 14064]
[New LWP 14065]
[New LWP 14066]
[New LWP 14069]
[New LWP 14070]
[New LWP 14071]
[New LWP 14072]
[New LWP 14073]
[New LWP 14074]
[New LWP 14075]
[New LWP 14076]
[New LWP 14077]
[New LWP 14078]
[New LWP 14079]
[New LWP 14080]
[New LWP 14081]
[New LWP 14082]
[New LWP 14083]
[New LWP 14085]
[New LWP 14087]
[New LWP 14088]
[New LWP 14089]
[New LWP 14092]
[New LWP 14093]
[New LWP 14094]
[New LWP 14095]
[New LWP 14096]
[New LWP 14097]
[New LWP 14098]
[New LWP 14099]
[New LWP 14100]
[New LWP 14101]
[New LWP 14102]
[New LWP 14103]
[New LWP 14104]
[New LWP 14105]
[New LWP 14106]
[New LWP 14107]
[New LWP 14108]
[New LWP 14109]
[New LWP 14110]
[New LWP 14111]
[New LWP 14112]
[New LWP 14113]
[New LWP 14114]
[New LWP 14115]
[New LWP 14116]
[New LWP 14117]
[New LWP 14118]
[New LWP 14119]
[New LWP 14120]
[New LWP 14121]
[New LWP 14122]
[New LWP 14123]
[New LWP 14124]
[New LWP 14125]
[New LWP 14126]
[New LWP 14127]
[New LWP 14128]
[New LWP 14129]
[New LWP 14130]
[New LWP 14131]
[New LWP 14132]
[New LWP 14133]
[New LWP 14134]
[New LWP 14135]
[New LWP 14136]
[New LWP 14137]
[New LWP 14138]
[New LWP 14139]
[New LWP 14142]
[New LWP 14143]
[New LWP 14144]
[New LWP 14145]
[New LWP 14146]
[New LWP 14147]
[New LWP 14148]
[New LWP 14149]
[New LWP 14150]
[New LWP 14151]
[New LWP 14152]
[New LWP 14153]
[New LWP 14154]
[New LWP 14155]
[New LWP 14156]
[New LWP 14157]
[New LWP 14158]
[New LWP 14159]
[New LWP 14160]
[New LWP 14161]
[New LWP 14162]
[New LWP 14163]
[New LWP 14164]
[New LWP 14165]
[New LWP 14166]
[New LWP 14167]
[New LWP 14168]
[New LWP 14169]
[New LWP 14170]
[New LWP 14171]
[New LWP 14172]
[New LWP 14173]
[New LWP 14174]
[New LWP 14175]
[New LWP 14176]
[New LWP 14177]
[New LWP 14178]
[New LWP 14179]
[New LWP 14180]
[New LWP 14181]
[New LWP 14182]
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
0x00007fa7085c0d50 in ?? ()
Id Target Id Frame
* 1 LWP 14038 "kudu" 0x00007fa7085c0d50 in ?? ()
2 LWP 14045 "kudu" 0x00007fa7085bcfb9 in ?? ()
3 LWP 14046 "kudu" 0x00007fa7085bcfb9 in ?? ()
4 LWP 14047 "kudu" 0x00007fa7085bcfb9 in ?? ()
5 LWP 14048 "kernel-watcher-" 0x00007fa7085bcfb9 in ?? ()
6 LWP 14054 "ntp client-1405" 0x00007fa7085c09e2 in ?? ()
7 LWP 14055 "file cache-evic" 0x00007fa7085bcfb9 in ?? ()
8 LWP 14056 "sq_acceptor" 0x00007fa7012e2cb9 in ?? ()
9 LWP 14061 "rpc reactor-140" 0x00007fa7012efa47 in ?? ()
10 LWP 14062 "rpc reactor-140" 0x00007fa7012efa47 in ?? ()
11 LWP 14063 "rpc reactor-140" 0x00007fa7012efa47 in ?? ()
12 LWP 14064 "rpc reactor-140" 0x00007fa7012efa47 in ?? ()
13 LWP 14065 "MaintenanceMgr " 0x00007fa712067d89 in ?? ()
14 LWP 14066 "txn-status-mana" 0x00007fa7085bcfb9 in ?? ()
15 LWP 14069 "collect_and_rem" 0x00007fa7085bcfb9 in ?? ()
16 LWP 14070 "tc-session-exp-" 0x00007fa7085bcfb9 in ?? ()
17 LWP 14071 "rpc worker-1407" 0x00007fa7085bcad3 in ?? ()
18 LWP 14072 "rpc worker-1407" 0x00007fa7085bcad3 in ?? ()
19 LWP 14073 "rpc worker-1407" 0x00007fa7085bcad3 in ?? ()
20 LWP 14074 "rpc worker-1407" 0x00007fa7085bcad3 in ?? ()
21 LWP 14075 "rpc worker-1407" 0x00007fa7085bcad3 in ?? ()
22 LWP 14076 "rpc worker-1407" 0x00007fa7085bcad3 in ?? ()
23 LWP 14077 "rpc worker-1407" 0x00007fa7085bcad3 in ?? ()
24 LWP 14078 "rpc worker-1407" 0x00007fa7085bcad3 in ?? ()
25 LWP 14079 "rpc worker-1407" 0x00007fa7085bcad3 in ?? ()
26 LWP 14080 "rpc worker-1408" 0x00007fa7085bcad3 in ?? ()
27 LWP 14081 "rpc worker-1408" 0x00007fa7085bcad3 in ?? ()
28 LWP 14082 "rpc worker-1408" 0x00007fa7085bcad3 in ?? ()
29 LWP 14083 "rpc worker-1408" 0x00007fa7085bcad3 in ?? ()
30 LWP 14085 "rpc worker-1408" 0x00007fa7085bcad3 in ?? ()
31 LWP 14087 "rpc worker-1408" 0x00007fa7085bcad3 in ?? ()
32 LWP 14088 "rpc worker-1408" 0x00007fa7085bcad3 in ?? ()
33 LWP 14089 "rpc worker-1408" 0x00007fa7085bcad3 in ?? ()
34 LWP 14092 "rpc worker-1409" 0x00007fa7085bcad3 in ?? ()
35 LWP 14093 "rpc worker-1409" 0x00007fa7085bcad3 in ?? ()
36 LWP 14094 "rpc worker-1409" 0x00007fa7085bcad3 in ?? ()
37 LWP 14095 "rpc worker-1409" 0x00007fa7085bcad3 in ?? ()
38 LWP 14096 "rpc worker-1409" 0x00007fa7085bcad3 in ?? ()
39 LWP 14097 "rpc worker-1409" 0x00007fa7085bcad3 in ?? ()
40 LWP 14098 "rpc worker-1409" 0x00007fa7085bcad3 in ?? ()
41 LWP 14099 "rpc worker-1409" 0x00007fa7085bcad3 in ?? ()
42 LWP 14100 "rpc worker-1410" 0x00007fa7085bcad3 in ?? ()
43 LWP 14101 "rpc worker-1410" 0x00007fa7085bcad3 in ?? ()
44 LWP 14102 "rpc worker-1410" 0x00007fa7085bcad3 in ?? ()
45 LWP 14103 "rpc worker-1410" 0x00007fa7085bcad3 in ?? ()
46 LWP 14104 "rpc worker-1410" 0x00007fa7085bcad3 in ?? ()
47 LWP 14105 "rpc worker-1410" 0x00007fa7085bcad3 in ?? ()
48 LWP 14106 "rpc worker-1410" 0x00007fa7085bcad3 in ?? ()
49 LWP 14107 "rpc worker-1410" 0x00007fa7085bcad3 in ?? ()
50 LWP 14108 "rpc worker-1410" 0x00007fa7085bcad3 in ?? ()
51 LWP 14109 "rpc worker-1410" 0x00007fa7085bcad3 in ?? ()
52 LWP 14110 "rpc worker-1411" 0x00007fa7085bcad3 in ?? ()
53 LWP 14111 "rpc worker-1411" 0x00007fa7085bcad3 in ?? ()
54 LWP 14112 "rpc worker-1411" 0x00007fa7085bcad3 in ?? ()
55 LWP 14113 "rpc worker-1411" 0x00007fa7085bcad3 in ?? ()
56 LWP 14114 "rpc worker-1411" 0x00007fa7085bcad3 in ?? ()
57 LWP 14115 "rpc worker-1411" 0x00007fa7085bcad3 in ?? ()
58 LWP 14116 "rpc worker-1411" 0x00007fa7085bcad3 in ?? ()
59 LWP 14117 "rpc worker-1411" 0x00007fa7085bcad3 in ?? ()
60 LWP 14118 "rpc worker-1411" 0x00007fa7085bcad3 in ?? ()
61 LWP 14119 "rpc worker-1411" 0x00007fa7085bcad3 in ?? ()
62 LWP 14120 "rpc worker-1412" 0x00007fa7085bcad3 in ?? ()
63 LWP 14121 "rpc worker-1412" 0x00007fa7085bcad3 in ?? ()
64 LWP 14122 "rpc worker-1412" 0x00007fa7085bcad3 in ?? ()
65 LWP 14123 "rpc worker-1412" 0x00007fa7085bcad3 in ?? ()
66 LWP 14124 "rpc worker-1412" 0x00007fa7085bcad3 in ?? ()
67 LWP 14125 "rpc worker-1412" 0x00007fa7085bcad3 in ?? ()
68 LWP 14126 "rpc worker-1412" 0x00007fa7085bcad3 in ?? ()
69 LWP 14127 "rpc worker-1412" 0x00007fa7085bcad3 in ?? ()
70 LWP 14128 "rpc worker-1412" 0x00007fa7085bcad3 in ?? ()
71 LWP 14129 "rpc worker-1412" 0x00007fa7085bcad3 in ?? ()
72 LWP 14130 "rpc worker-1413" 0x00007fa7085bcad3 in ?? ()
73 LWP 14131 "rpc worker-1413" 0x00007fa7085bcad3 in ?? ()
74 LWP 14132 "rpc worker-1413" 0x00007fa7085bcad3 in ?? ()
75 LWP 14133 "rpc worker-1413" 0x00007fa7085bcad3 in ?? ()
76 LWP 14134 "rpc worker-1413" 0x00007fa7085bcad3 in ?? ()
77 LWP 14135 "rpc worker-1413" 0x00007fa7085bcad3 in ?? ()
78 LWP 14136 "rpc worker-1413" 0x00007fa7085bcad3 in ?? ()
79 LWP 14137 "rpc worker-1413" 0x00007fa7085bcad3 in ?? ()
80 LWP 14138 "rpc worker-1413" 0x00007fa7085bcad3 in ?? ()
81 LWP 14139 "rpc worker-1413" 0x00007fa7085bcad3 in ?? ()
82 LWP 14142 "rpc worker-1414" 0x00007fa7085bcad3 in ?? ()
83 LWP 14143 "rpc worker-1414" 0x00007fa7085bcad3 in ?? ()
84 LWP 14144 "rpc worker-1414" 0x00007fa7085bcad3 in ?? ()
85 LWP 14145 "rpc worker-1414" 0x00007fa7085bcad3 in ?? ()
86 LWP 14146 "rpc worker-1414" 0x00007fa7085bcad3 in ?? ()
87 LWP 14147 "rpc worker-1414" 0x00007fa7085bcad3 in ?? ()
88 LWP 14148 "rpc worker-1414" 0x00007fa7085bcad3 in ?? ()
89 LWP 14149 "rpc worker-1414" 0x00007fa7085bcad3 in ?? ()
90 LWP 14150 "rpc worker-1415" 0x00007fa7085bcad3 in ?? ()
91 LWP 14151 "rpc worker-1415" 0x00007fa7085bcad3 in ?? ()
92 LWP 14152 "rpc worker-1415" 0x00007fa7085bcad3 in ?? ()
93 LWP 14153 "rpc worker-1415" 0x00007fa7085bcad3 in ?? ()
94 LWP 14154 "rpc worker-1415" 0x00007fa7085bcad3 in ?? ()
95 LWP 14155 "rpc worker-1415" 0x00007fa7085bcad3 in ?? ()
96 LWP 14156 "rpc worker-1415" 0x00007fa7085bcad3 in ?? ()
97 LWP 14157 "rpc worker-1415" 0x00007fa7085bcad3 in ?? ()
98 LWP 14158 "rpc worker-1415" 0x00007fa7085bcad3 in ?? ()
99 LWP 14159 "rpc worker-1415" 0x00007fa7085bcad3 in ?? ()
100 LWP 14160 "rpc worker-1416" 0x00007fa7085bcad3 in ?? ()
101 LWP 14161 "rpc worker-1416" 0x00007fa7085bcad3 in ?? ()
102 LWP 14162 "rpc worker-1416" 0x00007fa7085bcad3 in ?? ()
103 LWP 14163 "rpc worker-1416" 0x00007fa7085bcad3 in ?? ()
104 LWP 14164 "rpc worker-1416" 0x00007fa7085bcad3 in ?? ()
105 LWP 14165 "rpc worker-1416" 0x00007fa7085bcad3 in ?? ()
106 LWP 14166 "rpc worker-1416" 0x00007fa7085bcad3 in ?? ()
107 LWP 14167 "rpc worker-1416" 0x00007fa7085bcad3 in ?? ()
108 LWP 14168 "rpc worker-1416" 0x00007fa7085bcad3 in ?? ()
109 LWP 14169 "rpc worker-1416" 0x00007fa7085bcad3 in ?? ()
110 LWP 14170 "rpc worker-1417" 0x00007fa7085bcad3 in ?? ()
111 LWP 14171 "rpc worker-1417" 0x00007fa7085bcad3 in ?? ()
112 LWP 14172 "rpc worker-1417" 0x00007fa7085bcad3 in ?? ()
113 LWP 14173 "rpc worker-1417" 0x00007fa7085bcad3 in ?? ()
114 LWP 14174 "rpc worker-1417" 0x00007fa7085bcad3 in ?? ()
115 LWP 14175 "rpc worker-1417" 0x00007fa7085bcad3 in ?? ()
116 LWP 14176 "rpc worker-1417" 0x00007fa7085bcad3 in ?? ()
117 LWP 14177 "diag-logger-141" 0x00007fa70501a183 in ?? ()
118 LWP 14178 "result-tracker-" 0x00007fa7085bcfb9 in ?? ()
119 LWP 14179 "excess-log-dele" 0x00007fa7085bcfb9 in ?? ()
120 LWP 14180 "acceptor-14180" 0x00007fa7012f10c7 in ?? ()
121 LWP 14181 "heartbeat-14181" 0x00007fa7085bcfb9 in ?? ()
122 LWP 14182 "maintenance_sch" 0x00007fa7085bcfb9 in ?? ()
Thread 122 (LWP 14182):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x00007fa6bd41a6e0 in ?? ()
#2 0x0000000000000368 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007fa6bd41a730 in ?? ()
#6 0x00000000000006d0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 14181):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x8272756aa36a08ac in ?? ()
#2 0x00000000000000d9 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000613000019404 in ?? ()
#5 0x00007fa6bdc425f0 in ?? ()
#6 0x00000000000001b3 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fa6bdc42610 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007fa6bdc42690 in ?? ()
#12 0x00000ff4d7b884d4 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 14180):
#0 0x00007fa7012f10c7 in ?? ()
#1 0x00006140000214a8 in ?? ()
#2 0x00007fa6be459d90 in ?? ()
#3 0x00007fa6be459dc0 in ?? ()
#4 0x00007fa6be459ec0 in ?? ()
#5 0x00007fa6be459db0 in ?? ()
#6 0x00007fa6be459e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b082b in __sanitizer::theDepot ()
#9 0x00007fa7051773a0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 14179):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x00007fa6bec72f40 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffc2cefd2f0 in ?? ()
#5 0x00007fa6bec72f30 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 14178):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x00007fa6bf48b100 in ?? ()
#2 0x00000000000000da in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007fa6bf48b0f0 in ?? ()
#6 0x00000000000001b4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 14177):
#0 0x00007fa70501a183 in ?? ()
#1 0x0000000041b58ab3 in ?? ()
#2 0x00007fa704aee484 in ?? ()
#3 0x00007fa705019df0 in ?? ()
#4 0x000000000097a1b1 in __sanitizer::theDepot ()
#5 0x4059000000000000 in ?? ()
#6 0x00007fa6bfca2ca8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 116 (LWP 14176):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0003fffc8 in ?? ()
#4 0x00007fa6c04cbe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6c04cbeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 14175):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0003f67d8 in ?? ()
#4 0x00007fa6c0ce3e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6c0ce3eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 14174):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 14173):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 14172):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 14171):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 14170):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 14169):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 14168):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 14167):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 14166):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 14165):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 14164):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 14163):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 14162):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 14161):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 14160):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 14159):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 14158):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 14157):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 14156):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 14155):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 14154):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 14153):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 14152):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 14151):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 14150):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 14149):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 14148):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 14147):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 14146):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 14145):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
W20250625 03:46:49.716277 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
Thread 84 (LWP 14144):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 14143):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 14142):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 14139):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 14138):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 14137):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 14136):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 14135):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 14134):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000cd0 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00025ffc8 in ?? ()
#4 0x00007fa6d490be90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6d490beb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 14133):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000477 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002567dc in ?? ()
#4 0x00007fa6d5123e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6d5123eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d000256790 in ?? ()
#9 0x00007fa7085bc770 in ?? ()
#10 0x00007fa6d5123eb0 in ?? ()
#11 0x00000ff4daa247d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 14132):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x000000000000093e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00024ffd8 in ?? ()
#4 0x00007fa6d593be90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6d593beb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 14131):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 14130):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 14129):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 14128):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 14127):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 14126):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 14125):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 14124):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 14123):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 14122):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 14121):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 14120):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 14119):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 14118):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 14117):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 14116):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 14115):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 14114):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001c6868 in ?? ()
#4 0x00007fa6de2d4e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6de2d4eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 14113):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 14112):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 14111):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 14110):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 14109):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 14108):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 14107):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 14106):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 14105):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 14104):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 14103):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 14102):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 14101):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 14100):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 14099):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 14098):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 14097):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 14096):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 14095):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 14094):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000001934 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000100058 in ?? ()
#4 0x00007fa6e84f6e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6e84f6eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 14093):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000001828 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000f6868 in ?? ()
#4 0x00007fa6e8d0ee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6e8d0eeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 14092):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000001348 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000e6878 in ?? ()
#4 0x00007fa6e9526e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6e9526eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 14089):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000c5f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000e007c in ?? ()
#4 0x00007fa6e9d3ee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6e9d3eeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000e0030 in ?? ()
#9 0x00007fa7085bc770 in ?? ()
#10 0x00007fa6e9d3eeb0 in ?? ()
#11 0x00000ff4dd3a7dd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 32 (LWP 14088):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x00000000000012cd in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000b67dc in ?? ()
#4 0x00007fa6ea556e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6ea556eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000b6790 in ?? ()
#9 0x00007fa7085bc770 in ?? ()
#10 0x00007fa6ea556eb0 in ?? ()
#11 0x00000ff4dd4aadd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 31 (LWP 14087):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000001590 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000acfe8 in ?? ()
#4 0x00007fa6fa2b7e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6fa2b7eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 30 (LWP 14085):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x000000000000172f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007fa6ead7ee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa6ead7eeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007fa7085bc770 in ?? ()
#10 0x00007fa6ead7eeb0 in ?? ()
#11 0x00000ff4dd5afdd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 29 (LWP 14083):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 14082):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 14081):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 14080):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 14079):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 14078):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 14077):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 14076):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 14075):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 14074):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 14073):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 14072):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 14071):
#0 0x00007fa7085bcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 14070):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x00007fa6f2828cc0 in ?? ()
#2 0x0000000000000016 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a560 in ?? ()
#5 0x00007fa6f2828cb0 in ?? ()
#6 0x000000000000002c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 14069):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 14066):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x00007fa6f40af240 in ?? ()
#2 0x0000000000000016 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d1a8 in ?? ()
#5 0x00007fa6f40af230 in ?? ()
#6 0x000000000000002c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 14065):
#0 0x00007fa712067d89 in ?? ()
#1 0x0000000041b58ab3 in ?? ()
#2 0x00007fa711926ece in ?? ()
#3 0x00007fa712067c80 in ?? ()
#4 0x00007fa7120574c2 in ?? ()
#5 0x0000000045e0360e in ?? ()
#6 0x00007fa7118ef926 in ?? ()
#7 0x00007fa712057220 in ?? ()
#8 0x0000000000a751bd in __sanitizer::theDepot ()
#9 0x00007fa704c4f4c0 in ?? ()
#10 0x00007fa6f48abf40 in ?? ()
#11 0x00007fa6f48abe30 in ?? ()
#12 0x00007fa6f48abdc0 in ?? ()
#13 0x00007fa6f48abf40 in ?? ()
#14 0x00007fa6f48abe40 in ?? ()
#15 0x00007fa6f48abfd0 in ?? ()
#16 0x00007fa7120503a9 in ?? ()
#17 0x0000000045e0360e in ?? ()
#18 0x00007fa6f48abf60 in ?? ()
#19 0x00006120002ad250 in ?? ()
#20 0x00007fa6f48abf30 in ?? ()
#21 0x00000ff4de9157d0 in ?? ()
#22 0x00007fa6f48abe80 in ?? ()
#23 0x00007fa6f48abfd0 in ?? ()
#24 0x00007fa712052060 in ?? ()
#25 0x0000000041b58ab3 in ?? ()
#26 0x00007fa711919cb2 in ?? ()
#27 0x00007fa712051ef0 in ?? ()
#28 0x00007fa6f48ac270 in ?? ()
#29 0x00007fa6f48ac280 in ?? ()
#30 0x000060200002e590 in ?? ()
#31 0x00000ff55e90d850 in ?? ()
#32 0x000000000094be83 in __sanitizer::theDepot ()
#33 0x00007fa7050553f9 in ?? ()
#34 0x0000000000a751bd in __sanitizer::theDepot ()
#35 0x00007fa7051b8400 in ?? ()
#36 0x00007fa7051bb6cc in ?? ()
#37 0x00007fa7051bb549 in ?? ()
#38 0x0000000000a751bd in __sanitizer::theDepot ()
#39 0x00007fa705197f1e in ?? ()
#40 0x00007fa7085b66db in ?? ()
#41 0x000000000094a359 in __sanitizer::theDepot ()
#42 0x00007fa711ff284d in ?? ()
#43 0x00007fa704c4f440 in ?? ()
#44 0x00000ff55e90d7bc in ?? ()
#45 0x00007fa6f48abde0 in ?? ()
#46 0x00007fa711ff1d1d in ?? ()
#47 0x00007fa6b89a0208 in ?? ()
#48 0x00007fa6f48abde0 in ?? ()
#49 0x000060300045a4f0 in ?? ()
#50 0x00000ff4de9157f4 in ?? ()
#51 0x00007fa6f48ac0b0 in ?? ()
#52 0x00007fa6f48ac2c8 in ?? ()
#53 0x0000000000000000 in ?? ()
Thread 12 (LWP 14064):
#0 0x00007fa7012efa47 in ?? ()
#1 0x00007fa6f50b2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007fa6f50b2310 in ?? ()
#4 0x00007fa6f50b2520 in ?? ()
#5 0x00007fa6f50b2360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007fa6f50b23e0 in ?? ()
#8 0x00007fa70396225d in ?? ()
#9 0x3fb24a7a3f6d4000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000098e00000 in ?? ()
#14 0x00007fa600000000 in ?? ()
#15 0x41da16db2f4cc7aa in ?? ()
#16 0x00000ff55ea0e47c in ?? ()
#17 0x00007fa6f50b23c0 in ?? ()
#18 0x00007fa703966ba3 in ?? ()
#19 0x00007fa6f50b2390 in ?? ()
#20 0x3fb24a7a3f6d4000 in ?? ()
#21 0x00000000f50b23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb24a7a3f6d4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 14063):
#0 0x00007fa7012efa47 in ?? ()
#1 0x00007fa6f58b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007fa6f58b3310 in ?? ()
#4 0x00007fa6f58b3520 in ?? ()
#5 0x00007fa6f58b3360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007fa6f58b33e0 in ?? ()
#8 0x00007fa70396225d in ?? ()
#9 0x3fb98b8af8470000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000d7000000 in ?? ()
#14 0x00007fa600000000 in ?? ()
#15 0x41da16db2f4cc7ab in ?? ()
#16 0x00000ff55eb0e67c in ?? ()
#17 0x00007fa6f58b33c0 in ?? ()
#18 0x00007fa703966ba3 in ?? ()
#19 0x00007fa6f58b3390 in ?? ()
#20 0x3fb98b8af8470000 in ?? ()
#21 0x00000000f58b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb98b8af8470000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 14062):
#0 0x00007fa7012efa47 in ?? ()
#1 0x00007fa6f60b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007fa6f60b4310 in ?? ()
#4 0x00007fa6f60b4520 in ?? ()
#5 0x00007fa6f60b4360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007fa6f60b43e0 in ?? ()
#8 0x00007fa70396225d in ?? ()
#9 0x3fb94a7026190000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x000000008e000000 in ?? ()
#14 0x00007fa600000000 in ?? ()
#15 0x41da16db2f4cc7ad in ?? ()
#16 0x00000ff55ec0e87c in ?? ()
#17 0x00007fa6f60b43c0 in ?? ()
#18 0x00007fa703966ba3 in ?? ()
#19 0x00007fa6f60b4390 in ?? ()
#20 0x3fb94a7026190000 in ?? ()
#21 0x00000000f60b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021098 in ?? ()
#24 0x3fb94a7026190000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 14061):
#0 0x00007fa7012efa47 in ?? ()
#1 0x00007fa6f80ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007fa6f80ad310 in ?? ()
#4 0x00007fa6f80ad520 in ?? ()
#5 0x00007fa6f80ad360 in ?? ()
#6 0x0000614000020e98 in ?? ()
#7 0x00007fa6f80ad3e0 in ?? ()
#8 0x00007fa70396225d in ?? ()
#9 0x3fb7922ac2ffc000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x0000000059800000 in ?? ()
#14 0x00007fa600000000 in ?? ()
#15 0x41da16db2f4cc7ad in ?? ()
#16 0x00000ff55f00da7c in ?? ()
#17 0x00007fa6f80ad3c0 in ?? ()
#18 0x00007fa703966ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 14056):
#0 0x00007fa7012e2cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fa6f9ab77c8 in ?? ()
#3 0x000060200001ccd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a8471 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 14055):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x00007fa6f92b60c0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007fa6f92b60b0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 14054):
#0 0x00007fa7085c09e2 in ?? ()
#1 0x00007fa6f8ab3ba0 in ?? ()
#2 0x00007fa6f8ab3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fa6f8ab3bf0 in ?? ()
#6 0x000000000177e760 in ?? ()
#7 0x00007fa6f8ab3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007fa6f8ab3c40 in ?? ()
#10 0x00000000008a2967 in __sanitizer::theDepot ()
#11 0x00007fa70ded3dcc in ?? ()
#12 0x00007fa70dec37c9 in ?? ()
#13 0x00007fa70dec7db6 in ?? ()
#14 0x00007fa70decc7f0 in ?? ()
#15 0x00007fa70decc619 in ?? ()
#16 0x0000000000a751bd in __sanitizer::theDepot ()
#17 0x00007fa705197f1e in ?? ()
#18 0x00007fa7085b66db in ?? ()
#19 0x00000ff55f14e684 in ?? ()
#20 0x00007fa6f8ab3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 14048):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x00007fa6faab8c80 in ?? ()
#2 0x0000000000000443 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007fa6faab8c70 in ?? ()
#6 0x0000000000000886 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 14047):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 14046):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 14045):
#0 0x00007fa7085bcfb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000009 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007fa6fc2bc450 in ?? ()
#6 0x0000000000000012 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 14038):
#0 0x00007fa7085c0d50 in ?? ()
#1 0x0000000000000000 in ?? ()
I20250625 03:46:49.989431 16733 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:50.000747 16733 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:50.018757 16733 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:46:50.018116 14301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:50.019023 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:46:50.020151 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:46:50.022974 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
W20250625 03:46:50.022059 14063 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:50.027745 14064 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 975eeb611762460bb2f09f5b4b71e3d4; no voters: 506230f3ca974bf796583a6be750e38f, 513d19d66cd54765940762233636fd31, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:46:50.036275 16733 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
************************* END STACKS ***************************
I20250625 03:46:50.143716 12990 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 513d19d66cd54765940762233636fd31 and pid 14657
I20250625 03:46:50.153374 16727 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 10 similar messages]
W20250625 03:46:52.206597 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
W20250625 03:46:53.364626 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
I20250625 03:46:53.536675 16741 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 8 similar messages]
I20250625 03:46:54.079960 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (4.11M/6.60M this segment, stats: ops{read=2450 overwritten=0 applied=2448 ignored=0} inserts{seen=122300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:46:55.086153 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
W20250625 03:46:55.919030 14229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.175.131:40037, user_credentials={real_user=slave}} blocked reactor thread for 43686.2us
I20250625 03:46:56.457795 14065 maintenance_manager.cc:643] P 975eeb611762460bb2f09f5b4b71e3d4: FlushMRSOp(9126b7f95c9d434cb308581a7ce5b908) complete. Timing: real 105.593s user 27.865s sys 1.321s Metrics: {"bytes_written":6006478,"cfile_init":1,"dirs.queue_time_us":7411,"dirs.run_cpu_time_us":536,"dirs.run_wall_time_us":548,"drs_written":1,"lbm_read_time_us":759,"lbm_reads_lt_1ms":4,"lbm_write_time_us":521522,"lbm_writes_1-10_ms":32,"lbm_writes_10-100_ms":13,"lbm_writes_lt_1ms":228,"peak_mem_usage":0,"rows_written":214850,"thread_start_us":28097,"threads_started":2,"wal-append.queue_time_us":20606}
I20250625 03:46:56.462397 14182 maintenance_manager.cc:419] P 975eeb611762460bb2f09f5b4b71e3d4: Scheduling UndoDeltaBlockGCOp(9126b7f95c9d434cb308581a7ce5b908): 2464600 bytes on disk
I20250625 03:46:56.464809 14065 maintenance_manager.cc:643] P 975eeb611762460bb2f09f5b4b71e3d4: UndoDeltaBlockGCOp(9126b7f95c9d434cb308581a7ce5b908) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":318,"lbm_reads_lt_1ms":4}
I20250625 03:46:56.618554 16743 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 8 similar messages]
W20250625 03:46:56.785863 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
W20250625 03:46:58.292455 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
[New LWP 14663]
[New LWP 14665]
[New LWP 14666]
[New LWP 14669]
[New LWP 14675]
[New LWP 14676]
[New LWP 14677]
[New LWP 14680]
[New LWP 14681]
[New LWP 14682]
[New LWP 14684]
[New LWP 14685]
[New LWP 14686]
[New LWP 14688]
[New LWP 14689]
[New LWP 14691]
[New LWP 14694]
[New LWP 14695]
[New LWP 14696]
[New LWP 14697]
[New LWP 14699]
[New LWP 14700]
[New LWP 14701]
[New LWP 14702]
[New LWP 14703]
[New LWP 14704]
[New LWP 14705]
[New LWP 14706]
[New LWP 14707]
[New LWP 14708]
[New LWP 14709]
[New LWP 14710]
[New LWP 14711]
[New LWP 14712]
[New LWP 14713]
[New LWP 14714]
[New LWP 14715]
[New LWP 14716]
[New LWP 14717]
[New LWP 14718]
[New LWP 14719]
[New LWP 14720]
[New LWP 14721]
[New LWP 14722]
[New LWP 14723]
[New LWP 14724]
[New LWP 14725]
[New LWP 14726]
[New LWP 14727]
[New LWP 14728]
[New LWP 14729]
[New LWP 14730]
[New LWP 14731]
[New LWP 14732]
[New LWP 14733]
[New LWP 14734]
[New LWP 14735]
[New LWP 14736]
[New LWP 14737]
[New LWP 14738]
[New LWP 14739]
[New LWP 14740]
[New LWP 14741]
[New LWP 14742]
[New LWP 14743]
[New LWP 14744]
[New LWP 14745]
[New LWP 14746]
[New LWP 14747]
[New LWP 14748]
[New LWP 14749]
[New LWP 14750]
[New LWP 14751]
[New LWP 14752]
[New LWP 14753]
[New LWP 14754]
[New LWP 14755]
[New LWP 14756]
[New LWP 14757]
[New LWP 14758]
[New LWP 14759]
[New LWP 14760]
[New LWP 14761]
[New LWP 14762]
[New LWP 14763]
[New LWP 14764]
[New LWP 14765]
[New LWP 14766]
[New LWP 14767]
[New LWP 14768]
[New LWP 14769]
[New LWP 14770]
[New LWP 14771]
[New LWP 14773]
[New LWP 14774]
[New LWP 14775]
[New LWP 14779]
[New LWP 14780]
[New LWP 14781]
[New LWP 14782]
[New LWP 14783]
[New LWP 14784]
[New LWP 14785]
[New LWP 14786]
[New LWP 14787]
[New LWP 14788]
[New LWP 14789]
[New LWP 14790]
[New LWP 14791]
[New LWP 14792]
[New LWP 14793]
[New LWP 14794]
[New LWP 14795]
[New LWP 14796]
[New LWP 14797]
[New LWP 14798]
[New LWP 14799]
[New LWP 14800]
[New LWP 14801]
[New LWP 14802]
[New LWP 14805]
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
0x00007f7928e66d50 in ?? ()
Id Target Id Frame
* 1 LWP 14657 "kudu" 0x00007f7928e66d50 in ?? ()
2 LWP 14663 "kudu" 0x00007f7928e62fb9 in ?? ()
3 LWP 14665 "kudu" 0x00007f7928e62fb9 in ?? ()
4 LWP 14666 "kudu" 0x00007f7928e62fb9 in ?? ()
5 LWP 14669 "kernel-watcher-" 0x00007f7928e62fb9 in ?? ()
6 LWP 14675 "ntp client-1467" 0x00007f7928e669e2 in ?? ()
7 LWP 14676 "file cache-evic" 0x00007f7928e62fb9 in ?? ()
8 LWP 14677 "sq_acceptor" 0x00007f7921b88cb9 in ?? ()
9 LWP 14680 "rpc reactor-146" 0x00007f7921b95a47 in ?? ()
10 LWP 14681 "rpc reactor-146" 0x00007f7921b95a47 in ?? ()
11 LWP 14682 "rpc reactor-146" 0x00007f7921b95a47 in ?? ()
12 LWP 14684 "rpc reactor-146" 0x00007f7921b95a47 in ?? ()
13 LWP 14685 "MaintenanceMgr " 0x000000000087c3ae in __sanitizer::theDepot ()
14 LWP 14686 "txn-status-mana" 0x00007f7928e62fb9 in ?? ()
15 LWP 14688 "collect_and_rem" 0x00007f7928e62fb9 in ?? ()
16 LWP 14689 "tc-session-exp-" 0x00007f7928e62fb9 in ?? ()
17 LWP 14691 "rpc worker-1469" 0x00007f7928e62ad3 in ?? ()
18 LWP 14694 "rpc worker-1469" 0x00007f7928e62ad3 in ?? ()
19 LWP 14695 "rpc worker-1469" 0x00007f7928e62ad3 in ?? ()
20 LWP 14696 "rpc worker-1469" 0x00007f7928e62ad3 in ?? ()
21 LWP 14697 "rpc worker-1469" 0x00007f7928e62ad3 in ?? ()
22 LWP 14699 "rpc worker-1469" 0x00007f7928e62ad3 in ?? ()
23 LWP 14700 "rpc worker-1470" 0x00007f7928e62ad3 in ?? ()
24 LWP 14701 "rpc worker-1470" 0x00007f7928e62ad3 in ?? ()
25 LWP 14702 "rpc worker-1470" 0x00007f7928e62ad3 in ?? ()
26 LWP 14703 "rpc worker-1470" 0x00007f7928e62ad3 in ?? ()
27 LWP 14704 "rpc worker-1470" 0x00007f7928e62ad3 in ?? ()
28 LWP 14705 "rpc worker-1470" 0x00007f7928e62ad3 in ?? ()
29 LWP 14706 "rpc worker-1470" 0x00007f7928e62ad3 in ?? ()
30 LWP 14707 "rpc worker-1470" 0x00007f7928e62ad3 in ?? ()
31 LWP 14708 "rpc worker-1470" 0x00007f7928e62ad3 in ?? ()
32 LWP 14709 "rpc worker-1470" 0x00007f7928e62ad3 in ?? ()
33 LWP 14710 "rpc worker-1471" 0x00007f7928e62ad3 in ?? ()
34 LWP 14711 "rpc worker-1471" 0x00007f7928e62ad3 in ?? ()
35 LWP 14712 "rpc worker-1471" 0x00007f7928e62ad3 in ?? ()
36 LWP 14713 "rpc worker-1471" 0x00007f7928e62ad3 in ?? ()
37 LWP 14714 "rpc worker-1471" 0x00007f7928e62ad3 in ?? ()
38 LWP 14715 "rpc worker-1471" 0x00007f7928e62ad3 in ?? ()
39 LWP 14716 "rpc worker-1471" 0x00007f7928e62ad3 in ?? ()
40 LWP 14717 "rpc worker-1471" 0x00007f7928e62ad3 in ?? ()
41 LWP 14718 "rpc worker-1471" 0x00007f7928e62ad3 in ?? ()
42 LWP 14719 "rpc worker-1471" 0x00007f7928e62ad3 in ?? ()
43 LWP 14720 "rpc worker-1472" 0x00007f7928e62ad3 in ?? ()
44 LWP 14721 "rpc worker-1472" 0x00007f7928e62ad3 in ?? ()
45 LWP 14722 "rpc worker-1472" 0x00007f7928e62ad3 in ?? ()
46 LWP 14723 "rpc worker-1472" 0x00007f7928e62ad3 in ?? ()
47 LWP 14724 "rpc worker-1472" 0x00007f7928e62ad3 in ?? ()
48 LWP 14725 "rpc worker-1472" 0x00007f7928e62ad3 in ?? ()
49 LWP 14726 "rpc worker-1472" 0x00007f7928e62ad3 in ?? ()
50 LWP 14727 "rpc worker-1472" 0x00007f7928e62ad3 in ?? ()
51 LWP 14728 "rpc worker-1472" 0x00007f7928e62ad3 in ?? ()
52 LWP 14729 "rpc worker-1472" 0x00007f7928e62ad3 in ?? ()
53 LWP 14730 "rpc worker-1473" 0x00007f7928e62ad3 in ?? ()
54 LWP 14731 "rpc worker-1473" 0x00007f7928e62ad3 in ?? ()
55 LWP 14732 "rpc worker-1473" 0x00007f7928e62ad3 in ?? ()
56 LWP 14733 "rpc worker-1473" 0x00007f7928e62ad3 in ?? ()
57 LWP 14734 "rpc worker-1473" 0x00007f7928e62ad3 in ?? ()
58 LWP 14735 "rpc worker-1473" 0x00007f7928e62ad3 in ?? ()
59 LWP 14736 "rpc worker-1473" 0x00007f7928e62ad3 in ?? ()
60 LWP 14737 "rpc worker-1473" 0x00007f7928e62ad3 in ?? ()
61 LWP 14738 "rpc worker-1473" 0x00007f7928e62ad3 in ?? ()
62 LWP 14739 "rpc worker-1473" 0x00007f7928e62ad3 in ?? ()
63 LWP 14740 "rpc worker-1474" 0x00007f7928e62ad3 in ?? ()
64 LWP 14741 "rpc worker-1474" 0x00007f7928e62ad3 in ?? ()
65 LWP 14742 "rpc worker-1474" 0x00007f7928e62ad3 in ?? ()
66 LWP 14743 "rpc worker-1474" 0x00007f7928e62ad3 in ?? ()
67 LWP 14744 "rpc worker-1474" 0x00007f7928e62ad3 in ?? ()
68 LWP 14745 "rpc worker-1474" 0x00007f7928e62ad3 in ?? ()
69 LWP 14746 "rpc worker-1474" 0x00007f7928e62ad3 in ?? ()
70 LWP 14747 "rpc worker-1474" 0x00007f7928e62ad3 in ?? ()
71 LWP 14748 "rpc worker-1474" 0x00007f7928e62ad3 in ?? ()
72 LWP 14749 "rpc worker-1474" 0x00007f7928e62ad3 in ?? ()
73 LWP 14750 "rpc worker-1475" 0x00007f7928e62ad3 in ?? ()
74 LWP 14751 "rpc worker-1475" 0x00007f7928e62ad3 in ?? ()
75 LWP 14752 "rpc worker-1475" 0x00007f7928e62ad3 in ?? ()
76 LWP 14753 "rpc worker-1475" 0x00007f7928e62ad3 in ?? ()
77 LWP 14754 "rpc worker-1475" 0x00007f7928e62ad3 in ?? ()
78 LWP 14755 "rpc worker-1475" 0x00007f7928e62ad3 in ?? ()
79 LWP 14756 "rpc worker-1475" 0x00007f7928e62ad3 in ?? ()
80 LWP 14757 "rpc worker-1475" 0x00007f7928e62ad3 in ?? ()
81 LWP 14758 "rpc worker-1475" 0x00007f7928e62ad3 in ?? ()
82 LWP 14759 "rpc worker-1475" 0x00007f7928e62ad3 in ?? ()
83 LWP 14760 "rpc worker-1476" 0x00007f7928e62ad3 in ?? ()
84 LWP 14761 "rpc worker-1476" 0x00007f7928e62ad3 in ?? ()
85 LWP 14762 "rpc worker-1476" 0x00007f7928e62ad3 in ?? ()
86 LWP 14763 "rpc worker-1476" 0x00007f7928e62ad3 in ?? ()
87 LWP 14764 "rpc worker-1476" 0x00007f7928e62ad3 in ?? ()
88 LWP 14765 "rpc worker-1476" 0x00007f7928e62ad3 in ?? ()
89 LWP 14766 "rpc worker-1476" 0x00007f7928e62ad3 in ?? ()
90 LWP 14767 "rpc worker-1476" 0x00007f7928e62ad3 in ?? ()
91 LWP 14768 "rpc worker-1476" 0x00007f7928e62ad3 in ?? ()
92 LWP 14769 "rpc worker-1476" 0x00007f7928e62ad3 in ?? ()
93 LWP 14770 "rpc worker-1477" 0x00007f7928e62ad3 in ?? ()
94 LWP 14771 "rpc worker-1477" 0x00007f7928e62ad3 in ?? ()
95 LWP 14773 "rpc worker-1477" 0x00007f7928e62ad3 in ?? ()
96 LWP 14774 "rpc worker-1477" 0x00007f7928e62ad3 in ?? ()
97 LWP 14775 "rpc worker-1477" 0x00007f7928e62ad3 in ?? ()
98 LWP 14779 "rpc worker-1477" 0x00007f7928e62ad3 in ?? ()
99 LWP 14780 "rpc worker-1478" 0x00007f7928e62ad3 in ?? ()
100 LWP 14781 "rpc worker-1478" 0x00007f7928e62ad3 in ?? ()
101 LWP 14782 "rpc worker-1478" 0x00007f7928e62ad3 in ?? ()
102 LWP 14783 "rpc worker-1478" 0x00007f7928e62ad3 in ?? ()
103 LWP 14784 "rpc worker-1478" 0x00007f7928e62ad3 in ?? ()
104 LWP 14785 "rpc worker-1478" 0x00007f7928e62ad3 in ?? ()
105 LWP 14786 "rpc worker-1478" 0x00007f7928e62ad3 in ?? ()
106 LWP 14787 "rpc worker-1478" 0x00007f7928e62ad3 in ?? ()
107 LWP 14788 "rpc worker-1478" 0x00007f7928e62ad3 in ?? ()
108 LWP 14789 "rpc worker-1478" 0x00007f7928e62ad3 in ?? ()
109 LWP 14790 "rpc worker-1479" 0x00007f7928e62ad3 in ?? ()
110 LWP 14791 "rpc worker-1479" 0x00007f7928e62ad3 in ?? ()
111 LWP 14792 "rpc worker-1479" 0x00007f7928e62ad3 in ?? ()
112 LWP 14793 "rpc worker-1479" 0x00007f7928e62ad3 in ?? ()
113 LWP 14794 "rpc worker-1479" 0x00007f7928e62ad3 in ?? ()
114 LWP 14795 "rpc worker-1479" 0x00007f7928e62ad3 in ?? ()
115 LWP 14796 "rpc worker-1479" 0x00007f7928e62ad3 in ?? ()
116 LWP 14797 "rpc worker-1479" 0x00007f7928e62ad3 in ?? ()
117 LWP 14798 "diag-logger-147" 0x00007f7928e62fb9 in ?? ()
118 LWP 14799 "result-tracker-" 0x00007f7928e62fb9 in ?? ()
119 LWP 14800 "excess-log-dele" 0x00007f7928e62fb9 in ?? ()
120 LWP 14801 "acceptor-14801" 0x00007f7921b970c7 in ?? ()
121 LWP 14802 "heartbeat-14802" 0x00007f7928e62fb9 in ?? ()
122 LWP 14805 "maintenance_sch" 0x00007f7928e62fb9 in ?? ()
Thread 122 (LWP 14805):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x00007f78ddcd06e0 in ?? ()
#2 0x000000000000031b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f78ddcd0730 in ?? ()
#6 0x0000000000000636 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 14802):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x8272756aa36a08ac in ?? ()
#2 0x00000000000000c4 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007f78de4e85f0 in ?? ()
#6 0x0000000000000188 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 14801):
#0 0x00007f7921b970c7 in ?? ()
#1 0x00006140000212a8 in ?? ()
#2 0x00007f78decffd90 in ?? ()
#3 0x00007f78decffdc0 in ?? ()
#4 0x00007f78decffec0 in ?? ()
#5 0x00007f78decffdb0 in ?? ()
#6 0x00007f78decffe20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b082b in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 14800):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x00007f78df518f40 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffd05a57c30 in ?? ()
#5 0x00007f78df518f30 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 14799):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x00007f78dfd31100 in ?? ()
#2 0x00000000000000c7 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f78dfd310f0 in ?? ()
#6 0x000000000000018e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 14798):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 14797):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x000000000000000c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0003c6808 in ?? ()
#4 0x00007f78e0d71e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f78e0d71eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 14796):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0003b6818 in ?? ()
#4 0x00007f78e1589e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f78e1589eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 14795):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 14794):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 14793):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 14792):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 14791):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 14790):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 14789):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 14788):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 14787):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 14786):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 14785):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 14784):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 14783):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 14782):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 14781):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 14780):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 14779):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 14775):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 14774):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 14773):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 14771):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 14770):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 14769):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 14768):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 14767):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 14766):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 14765):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 14764):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 14763):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 14762):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 14761):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 14760):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 14759):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 14758):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 14757):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 14756):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 14755):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 14754):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 14753):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000419 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f78f5324e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f78f5324eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f7928e62770 in ?? ()
#10 0x00007f78f5324eb0 in ?? ()
#11 0x00000fef1ea649d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 14752):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x00000000000006ea in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007f78f5b3ce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f78f5b3ceb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 14751):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x000000000000067a in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a0088 in ?? ()
#4 0x00007f78f6354e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f78f6354eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 14750):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 14749):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 14748):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 14747):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 14746):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 14745):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 14744):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 14743):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 14742):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 14741):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 14740):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 14739):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 14738):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 14737):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 14736):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 14735):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 14734):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 14733):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000007 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f78ff505e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f78ff505eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f7928e62770 in ?? ()
#10 0x00007f78ff505eb0 in ?? ()
#11 0x00000fef1fea0bd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 14732):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 14731):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 14730):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 14729):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 14728):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 14727):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 14726):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 14725):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 14724):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 14723):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 14722):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 14721):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 14720):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 14719):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 14718):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 14717):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 14716):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 14715):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 14714):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 14713):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000007 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f79096e6e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f79096e6eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f7928e62770 in ?? ()
#10 0x00007f79096e6eb0 in ?? ()
#11 0x00000fef212dcdd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 14712):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 14711):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 14710):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 14709):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 14708):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 14707):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 14706):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 14705):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 14704):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 14703):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 14702):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 14701):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 14700):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 14699):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 14697):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 14696):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 14695):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 14694):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 14691):
#0 0x00007f7928e62ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 14689):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x00007f79130decc0 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a560 in ?? ()
#5 0x00007f79130decb0 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 14688):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 14686):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x00007f7914963240 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d1a8 in ?? ()
#5 0x00007f7914963230 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 14685):
#0 0x000000000087c3ae in __sanitizer::theDepot ()
#1 0x00007f791497e0e0 in ?? ()
#2 0x0000000000000073 in ?? ()
#3 0x00007f7915186950 in ?? ()
#4 0x00007f79149897b0 in ?? ()
#5 0x0000000000000004 in ?? ()
#6 0x0000000000ed8b28 in ?? ()
#7 0x00007f7915186a10 in ?? ()
#8 0x00007f791497e000 in ?? ()
#9 0x000000000e666661 in ?? ()
#10 0x0000000000100000 in ?? ()
#11 0x0000604000623510 in ?? ()
#12 0x000000000087d55e in __sanitizer::theDepot ()
#13 0x00007f7915186a90 in ?? ()
#14 0x00007f793082900f in ?? ()
#15 0x0000000000000008 in ?? ()
#16 0x00007f793069a620 in ?? ()
#17 0x00007f7930828cd0 in ?? ()
#18 0x0000000000a7774c in __sanitizer::theDepot ()
#19 0x00007f7900000002 in ?? ()
#20 0x00007f7900000000 in ?? ()
#21 0x000000000000000e in ?? ()
#22 0x0000624000c98000 in ?? ()
#23 0x0000624000bc8000 in ?? ()
#24 0x0000000000e23200 in getTypeCacheHashTableBucket(unsigned long)::__ubsan_vptr_hash_set ()
#25 0x00007f7915186a70 in ?? ()
#26 0x0000000000008000 in ?? ()
#27 0x00007f7915186a10 in ?? ()
#28 0x00007f791497e000 in ?? ()
#29 0x000062d001a183f0 in ?? ()
#30 0x0000000000100000 in ?? ()
#31 0x0000604000623510 in ?? ()
#32 0x0000000000881a9f in __sanitizer::theDepot ()
#33 0x00007f791498d000 in ?? ()
#34 0x00007f7915186a10 in ?? ()
#35 0x00007f791497e0e0 in ?? ()
#36 0x00007f7915186a10 in ?? ()
#37 0x00000fefa2a28d40 in ?? ()
#38 0x00007f7915186a10 in ?? ()
#39 0x00007f7915186a10 in ?? ()
#40 0x00007f7915187250 in ?? ()
#41 0x000062d001a18400 in ?? ()
#42 0x000000000000001e in ?? ()
#43 0x0000000000000001 in ?? ()
#44 0x00000000009057b1 in __sanitizer::theDepot ()
#45 0x00007f7915186a20 in ?? ()
#46 0x000000000000001e in ?? ()
#47 0x00000000009057e1 in __sanitizer::theDepot ()
#48 0x00007f7925918c10 in ?? ()
#49 0x00007f7925917731 in ?? ()
#50 0x00000000009f238b in __sanitizer::theDepot ()
#51 0x00000000009f22ec in __sanitizer::theDepot ()
#52 0x00000000009f228e in __sanitizer::theDepot ()
#53 0x00000000009f213b in __sanitizer::theDepot ()
#54 0x00000000009f209c in __sanitizer::theDepot ()
#55 0x00000000009f201c in __sanitizer::theDepot ()
#56 0x00000000009f1fc5 in __sanitizer::theDepot ()
#57 0x00000000009f1f81 in __sanitizer::theDepot ()
#58 0x00000000009f1eb1 in __sanitizer::theDepot ()
#59 0x00007f79259162ae in ?? ()
#60 0x00007f79259132d1 in ?? ()
#61 0x00007f7925912e3f in ?? ()
#62 0x00007f7932872120 in ?? ()
#63 0x00007f793286283f in ?? ()
#64 0x00007f793267525c in ?? ()
#65 0x00007f793267257d in ?? ()
#66 0x00007f793277bfe7 in ?? ()
#67 0x00007f79258f86c9 in ?? ()
#68 0x00007f79258fb563 in ?? ()
#69 0x00007f79258fb3f9 in ?? ()
#70 0x0000000000a751bd in __sanitizer::theDepot ()
#71 0x00007f7925a5e400 in ?? ()
#72 0x00007f7925a616cc in ?? ()
#73 0x00007f7925a61549 in ?? ()
#74 0x0000000000a751bd in __sanitizer::theDepot ()
#75 0x00007f7925a3df1e in ?? ()
#76 0x00007f7928e5c6db in ?? ()
#77 0x000061300010fd00 in ?? ()
#78 0x000061400031be40 in ?? ()
#79 0x0000000000000000 in ?? ()
Thread 12 (LWP 14684):
#0 0x00007f7921b95a47 in ?? ()
#1 0x00007f79159b2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f79159b2310 in ?? ()
#4 0x00007f79159b2520 in ?? ()
#5 0x00007f79159b2360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f79159b23e0 in ?? ()
#8 0x00007f792420825d in ?? ()
#9 0x3fb1829c0901c000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000030000000 in ?? ()
#14 0x00007f7900000000 in ?? ()
#15 0x41da16db2f4cc7ac in ?? ()
#16 0x00000fefa2b2e47c in ?? ()
#17 0x00007f79159b23c0 in ?? ()
#18 0x00007f792420cba3 in ?? ()
#19 0x00007f79159b2390 in ?? ()
#20 0x3fb1829c0901c000 in ?? ()
#21 0x00000000159b23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb1829c0901c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 14682):
#0 0x00007f7921b95a47 in ?? ()
#1 0x00007f79161b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f79161b3310 in ?? ()
#4 0x00007f79161b3520 in ?? ()
#5 0x00007f79161b3360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f79161b33e0 in ?? ()
#8 0x00007f792420825d in ?? ()
#9 0x3fb95dbbbef5c000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000e6200000 in ?? ()
#14 0x0000000300000000 in ?? ()
#15 0x41da16db2f4cc7a9 in ?? ()
#16 0xbeb0000000000000 in ?? ()
#17 0x00007f79161b33c0 in ?? ()
#18 0x00007f792420cba3 in ?? ()
#19 0x00007f79161b3390 in ?? ()
#20 0x3fb95dbbbef5c000 in ?? ()
#21 0x00000000161b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021098 in ?? ()
#24 0x3fb95dbbbef5c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 14681):
#0 0x00007f7921b95a47 in ?? ()
#1 0x00007f79169b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f79169b4310 in ?? ()
#4 0x00007f79169b4520 in ?? ()
#5 0x00007f79169b4360 in ?? ()
#6 0x0000614000020e98 in ?? ()
#7 0x00007f79169b43e0 in ?? ()
#8 0x00007f792420825d in ?? ()
#9 0x3fb969a43bad8000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000b4800000 in ?? ()
#14 0x00007f7900000000 in ?? ()
#15 0x41da16db2f4cc7b1 in ?? ()
#16 0x00000fefa2d2e87c in ?? ()
#17 0x00007f79169b43c0 in ?? ()
#18 0x00007f792420cba3 in ?? ()
#19 0x00007f79169b4390 in ?? ()
#20 0x3fb969a43bad8000 in ?? ()
#21 0x00000000169b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000020e98 in ?? ()
#24 0x3fb969a43bad8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 14680):
#0 0x00007f7921b95a47 in ?? ()
#1 0x00007f79189ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f79189ad310 in ?? ()
#4 0x00007f79189ad520 in ?? ()
#5 0x00007f79189ad360 in ?? ()
#6 0x0000614000020c98 in ?? ()
#7 0x00007f79189ad3e0 in ?? ()
#8 0x00007f792420825d in ?? ()
#9 0x3fb3d23e61c94000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000a7f00000 in ?? ()
#14 0x00007f7900000000 in ?? ()
#15 0x41da16db2f4cc7aa in ?? ()
#16 0x00000fefa312da7c in ?? ()
#17 0x00007f79189ad3c0 in ?? ()
#18 0x00007f792420cba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 14677):
#0 0x00007f7921b88cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f791a3b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a8471 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 14676):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x00007f7919bb60c0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007f7919bb60b0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 14675):
#0 0x00007f7928e669e2 in ?? ()
#1 0x00007f79193b3ba0 in ?? ()
#2 0x00007f79193b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f79193b3bf0 in ?? ()
#6 0x000000000177e760 in ?? ()
#7 0x00007f79193b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f79193b3c40 in ?? ()
#10 0x00000000008a2967 in __sanitizer::theDepot ()
#11 0x00007f792e779dcc in ?? ()
#12 0x00007f792e7697c9 in ?? ()
#13 0x00007f792e76ddb6 in ?? ()
#14 0x00007f792e7727f0 in ?? ()
#15 0x00007f792e772619 in ?? ()
#16 0x0000000000a751bd in __sanitizer::theDepot ()
#17 0x00007f7925a3df1e in ?? ()
#18 0x00007f7928e5c6db in ?? ()
#19 0x00000fefa326e684 in ?? ()
#20 0x00007f79193b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 14669):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x00007f791b3b8c80 in ?? ()
#2 0x00000000000003d5 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007f791b3b8c70 in ?? ()
#6 0x00000000000007aa in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 14666):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 14665):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 14663):
#0 0x00007f7928e62fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f791cbbc450 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 14657):
#0 0x00007f7928e66d50 in ?? ()
#1 0x0000000000000000 in ?? ()
I20250625 03:46:58.944401 16755 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:46:58.956425 16755 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:46:58.970458 16755 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:46:58.971243 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:46:58.975337 14301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:46:58.980557 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:46:58.983851 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:46:58.984690 14682 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:46:58.985013 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:46:58.986274 16755 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
************************* END STACKS ***************************
I20250625 03:46:59.038913 12990 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID d275425c3ef443f88abb12d24f4c2f2f and pid 14819
I20250625 03:46:59.092741 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (4.45M/6.60M this segment, stats: ops{read=2651 overwritten=0 applied=2651 ignored=0} inserts{seen=132450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:46:59.596977 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 256: this message will repeat every 5th retry.
I20250625 03:46:59.644913 16751 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 10 similar messages]
W20250625 03:47:00.852913 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 261: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
W20250625 03:47:02.454764 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 266: this message will repeat every 5th retry.
I20250625 03:47:02.896064 16760 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 10 similar messages]
I20250625 03:47:04.100731 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (4.98M/6.60M this segment, stats: ops{read=2967 overwritten=0 applied=2967 ignored=0} inserts{seen=148250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:47:04.101850 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 271: this message will repeat every 5th retry.
[New LWP 14822]
[New LWP 14823]
[New LWP 14824]
[New LWP 14825]
[New LWP 14831]
[New LWP 14833]
[New LWP 14834]
[New LWP 14838]
[New LWP 14839]
[New LWP 14840]
[New LWP 14841]
[New LWP 14842]
[New LWP 14843]
[New LWP 14845]
[New LWP 14846]
[New LWP 14847]
[New LWP 14848]
[New LWP 14849]
[New LWP 14850]
[New LWP 14851]
[New LWP 14852]
[New LWP 14853]
[New LWP 14854]
[New LWP 14855]
[New LWP 14856]
[New LWP 14857]
[New LWP 14858]
[New LWP 14859]
[New LWP 14860]
[New LWP 14861]
[New LWP 14862]
[New LWP 14863]
[New LWP 14864]
[New LWP 14865]
[New LWP 14866]
[New LWP 14867]
[New LWP 14868]
[New LWP 14869]
[New LWP 14870]
[New LWP 14871]
[New LWP 14872]
[New LWP 14873]
[New LWP 14874]
[New LWP 14875]
[New LWP 14876]
[New LWP 14877]
[New LWP 14878]
[New LWP 14879]
[New LWP 14880]
[New LWP 14881]
[New LWP 14882]
[New LWP 14883]
[New LWP 14884]
[New LWP 14885]
[New LWP 14886]
[New LWP 14887]
[New LWP 14888]
[New LWP 14889]
[New LWP 14890]
[New LWP 14891]
[New LWP 14892]
[New LWP 14893]
[New LWP 14894]
[New LWP 14895]
[New LWP 14896]
[New LWP 14897]
[New LWP 14898]
[New LWP 14899]
[New LWP 14900]
[New LWP 14901]
[New LWP 14902]
[New LWP 14903]
[New LWP 14904]
[New LWP 14905]
[New LWP 14906]
[New LWP 14907]
[New LWP 14908]
[New LWP 14909]
[New LWP 14910]
[New LWP 14911]
[New LWP 14912]
[New LWP 14913]
[New LWP 14914]
[New LWP 14916]
[New LWP 14917]
[New LWP 14918]
[New LWP 14920]
[New LWP 14921]
[New LWP 14922]
[New LWP 14923]
[New LWP 14924]
[New LWP 14925]
[New LWP 14926]
[New LWP 14927]
[New LWP 14928]
[New LWP 14929]
[New LWP 14930]
[New LWP 14931]
[New LWP 14932]
[New LWP 14933]
[New LWP 14934]
[New LWP 14935]
[New LWP 14936]
[New LWP 14937]
[New LWP 14938]
[New LWP 14939]
[New LWP 14940]
[New LWP 14941]
[New LWP 14942]
[New LWP 14943]
[New LWP 14944]
[New LWP 14945]
[New LWP 14946]
[New LWP 14947]
[New LWP 14948]
[New LWP 14949]
[New LWP 14950]
[New LWP 14951]
[New LWP 14952]
[New LWP 14953]
[New LWP 14956]
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
0x00007f2f03e22d50 in ?? ()
Id Target Id Frame
* 1 LWP 14819 "kudu" 0x00007f2f03e22d50 in ?? ()
2 LWP 14822 "kudu" 0x00007f2f03e1efb9 in ?? ()
3 LWP 14823 "kudu" 0x00007f2f03e1efb9 in ?? ()
4 LWP 14824 "kudu" 0x00007f2f03e1efb9 in ?? ()
5 LWP 14825 "kernel-watcher-" 0x00007f2f03e1efb9 in ?? ()
6 LWP 14831 "ntp client-1483" 0x00007f2f03e229e2 in ?? ()
7 LWP 14833 "file cache-evic" 0x00007f2f03e1efb9 in ?? ()
8 LWP 14834 "sq_acceptor" 0x00007f2efcb44cb9 in ?? ()
9 LWP 14838 "rpc reactor-148" 0x00007f2efcb51a47 in ?? ()
10 LWP 14839 "rpc reactor-148" 0x00007f2efcb51a47 in ?? ()
11 LWP 14840 "rpc reactor-148" 0x00007f2efcb51a47 in ?? ()
12 LWP 14841 "rpc reactor-148" 0x00007f2efcb51a47 in ?? ()
13 LWP 14842 "MaintenanceMgr " 0x00000000009f49c8 in __sanitizer::theDepot ()
14 LWP 14843 "txn-status-mana" 0x00007f2f03e1efb9 in ?? ()
15 LWP 14845 "collect_and_rem" 0x00007f2f03e1efb9 in ?? ()
16 LWP 14846 "tc-session-exp-" 0x00007f2f03e1efb9 in ?? ()
17 LWP 14847 "rpc worker-1484" 0x00007f2f03e1ead3 in ?? ()
18 LWP 14848 "rpc worker-1484" 0x00007f2f03e1ead3 in ?? ()
19 LWP 14849 "rpc worker-1484" 0x00007f2f03e1ead3 in ?? ()
20 LWP 14850 "rpc worker-1485" 0x00007f2f03e1ead3 in ?? ()
21 LWP 14851 "rpc worker-1485" 0x00007f2f03e1ead3 in ?? ()
22 LWP 14852 "rpc worker-1485" 0x00007f2f03e1ead3 in ?? ()
23 LWP 14853 "rpc worker-1485" 0x00007f2f03e1ead3 in ?? ()
24 LWP 14854 "rpc worker-1485" 0x00007f2f03e1ead3 in ?? ()
25 LWP 14855 "rpc worker-1485" 0x00007f2f03e1ead3 in ?? ()
26 LWP 14856 "rpc worker-1485" 0x00007f2f03e1ead3 in ?? ()
27 LWP 14857 "rpc worker-1485" 0x00007f2f03e1ead3 in ?? ()
28 LWP 14858 "rpc worker-1485" 0x00007f2f03e1ead3 in ?? ()
29 LWP 14859 "rpc worker-1485" 0x00007f2f03e1ead3 in ?? ()
30 LWP 14860 "rpc worker-1486" 0x00007f2f03e1ead3 in ?? ()
31 LWP 14861 "rpc worker-1486" 0x00007f2f03e1ead3 in ?? ()
32 LWP 14862 "rpc worker-1486" 0x00007f2f03e1ead3 in ?? ()
33 LWP 14863 "rpc worker-1486" 0x00007f2f03e1ead3 in ?? ()
34 LWP 14864 "rpc worker-1486" 0x00007f2f03e1ead3 in ?? ()
35 LWP 14865 "rpc worker-1486" 0x00007f2f03e1ead3 in ?? ()
36 LWP 14866 "rpc worker-1486" 0x00007f2f03e1ead3 in ?? ()
37 LWP 14867 "rpc worker-1486" 0x00007f2f03e1ead3 in ?? ()
38 LWP 14868 "rpc worker-1486" 0x00007f2f03e1ead3 in ?? ()
39 LWP 14869 "rpc worker-1486" 0x00007f2f03e1ead3 in ?? ()
40 LWP 14870 "rpc worker-1487" 0x00007f2f03e1ead3 in ?? ()
41 LWP 14871 "rpc worker-1487" 0x00007f2f03e1ead3 in ?? ()
42 LWP 14872 "rpc worker-1487" 0x00007f2f03e1ead3 in ?? ()
43 LWP 14873 "rpc worker-1487" 0x00007f2f03e1ead3 in ?? ()
44 LWP 14874 "rpc worker-1487" 0x00007f2f03e1ead3 in ?? ()
45 LWP 14875 "rpc worker-1487" 0x00007f2f03e1ead3 in ?? ()
46 LWP 14876 "rpc worker-1487" 0x00007f2f03e1ead3 in ?? ()
47 LWP 14877 "rpc worker-1487" 0x00007f2f03e1ead3 in ?? ()
48 LWP 14878 "rpc worker-1487" 0x00007f2f03e1ead3 in ?? ()
49 LWP 14879 "rpc worker-1487" 0x00007f2f03e1ead3 in ?? ()
50 LWP 14880 "rpc worker-1488" 0x00007f2f03e1ead3 in ?? ()
51 LWP 14881 "rpc worker-1488" 0x00007f2f03e1ead3 in ?? ()
52 LWP 14882 "rpc worker-1488" 0x00007f2f03e1ead3 in ?? ()
53 LWP 14883 "rpc worker-1488" 0x00007f2f03e1ead3 in ?? ()
54 LWP 14884 "rpc worker-1488" 0x00007f2f03e1ead3 in ?? ()
55 LWP 14885 "rpc worker-1488" 0x00007f2f03e1ead3 in ?? ()
56 LWP 14886 "rpc worker-1488" 0x00007f2f03e1ead3 in ?? ()
57 LWP 14887 "rpc worker-1488" 0x00007f2f03e1ead3 in ?? ()
58 LWP 14888 "rpc worker-1488" 0x00007f2f03e1ead3 in ?? ()
59 LWP 14889 "rpc worker-1488" 0x00007f2f03e1ead3 in ?? ()
60 LWP 14890 "rpc worker-1489" 0x00007f2f03e1ead3 in ?? ()
61 LWP 14891 "rpc worker-1489" 0x00007f2f03e1ead3 in ?? ()
62 LWP 14892 "rpc worker-1489" 0x00007f2f03e1ead3 in ?? ()
63 LWP 14893 "rpc worker-1489" 0x00007f2f03e1ead3 in ?? ()
64 LWP 14894 "rpc worker-1489" 0x00007f2f03e1ead3 in ?? ()
65 LWP 14895 "rpc worker-1489" 0x00007f2f03e1ead3 in ?? ()
66 LWP 14896 "rpc worker-1489" 0x00007f2f03e1ead3 in ?? ()
67 LWP 14897 "rpc worker-1489" 0x00007f2f03e1ead3 in ?? ()
68 LWP 14898 "rpc worker-1489" 0x00007f2f03e1ead3 in ?? ()
69 LWP 14899 "rpc worker-1489" 0x00007f2f03e1ead3 in ?? ()
70 LWP 14900 "rpc worker-1490" 0x00007f2f03e1ead3 in ?? ()
71 LWP 14901 "rpc worker-1490" 0x00007f2f03e1ead3 in ?? ()
72 LWP 14902 "rpc worker-1490" 0x00007f2f03e1ead3 in ?? ()
73 LWP 14903 "rpc worker-1490" 0x00007f2f03e1ead3 in ?? ()
74 LWP 14904 "rpc worker-1490" 0x00007f2f03e1ead3 in ?? ()
75 LWP 14905 "rpc worker-1490" 0x00007f2f03e1ead3 in ?? ()
76 LWP 14906 "rpc worker-1490" 0x00007f2f03e1ead3 in ?? ()
77 LWP 14907 "rpc worker-1490" 0x00007f2f03e1ead3 in ?? ()
78 LWP 14908 "rpc worker-1490" 0x00007f2f03e1ead3 in ?? ()
79 LWP 14909 "rpc worker-1490" 0x00007f2f03e1ead3 in ?? ()
80 LWP 14910 "rpc worker-1491" 0x00007f2f03e1ead3 in ?? ()
81 LWP 14911 "rpc worker-1491" 0x00007f2f03e1ead3 in ?? ()
82 LWP 14912 "rpc worker-1491" 0x00007f2f03e1ead3 in ?? ()
83 LWP 14913 "rpc worker-1491" 0x00007f2f03e1ead3 in ?? ()
84 LWP 14914 "rpc worker-1491" 0x00007f2f03e1ead3 in ?? ()
85 LWP 14916 "rpc worker-1491" 0x00007f2f03e1ead3 in ?? ()
86 LWP 14917 "rpc worker-1491" 0x00007f2f03e1ead3 in ?? ()
87 LWP 14918 "rpc worker-1491" 0x00007f2f03e1ead3 in ?? ()
88 LWP 14920 "rpc worker-1492" 0x00007f2f03e1ead3 in ?? ()
89 LWP 14921 "rpc worker-1492" 0x00007f2f03e1ead3 in ?? ()
90 LWP 14922 "rpc worker-1492" 0x00007f2f03e1ead3 in ?? ()
91 LWP 14923 "rpc worker-1492" 0x00007f2f03e1ead3 in ?? ()
92 LWP 14924 "rpc worker-1492" 0x00007f2f03e1ead3 in ?? ()
93 LWP 14925 "rpc worker-1492" 0x00007f2f03e1ead3 in ?? ()
94 LWP 14926 "rpc worker-1492" 0x00007f2f03e1ead3 in ?? ()
95 LWP 14927 "rpc worker-1492" 0x00007f2f03e1ead3 in ?? ()
96 LWP 14928 "rpc worker-1492" 0x00007f2f03e1ead3 in ?? ()
97 LWP 14929 "rpc worker-1492" 0x00007f2f03e1ead3 in ?? ()
98 LWP 14930 "rpc worker-1493" 0x00007f2f03e1ead3 in ?? ()
99 LWP 14931 "rpc worker-1493" 0x00007f2f03e1ead3 in ?? ()
100 LWP 14932 "rpc worker-1493" 0x00007f2f03e1ead3 in ?? ()
101 LWP 14933 "rpc worker-1493" 0x00007f2f03e1ead3 in ?? ()
102 LWP 14934 "rpc worker-1493" 0x00007f2f03e1ead3 in ?? ()
103 LWP 14935 "rpc worker-1493" 0x00007f2f03e1ead3 in ?? ()
104 LWP 14936 "rpc worker-1493" 0x00007f2f03e1ead3 in ?? ()
105 LWP 14937 "rpc worker-1493" 0x00007f2f03e1ead3 in ?? ()
106 LWP 14938 "rpc worker-1493" 0x00007f2f03e1ead3 in ?? ()
107 LWP 14939 "rpc worker-1493" 0x00007f2f03e1ead3 in ?? ()
108 LWP 14940 "rpc worker-1494" 0x00007f2f03e1ead3 in ?? ()
109 LWP 14941 "rpc worker-1494" 0x00007f2f03e1ead3 in ?? ()
110 LWP 14942 "rpc worker-1494" 0x00007f2f03e1ead3 in ?? ()
111 LWP 14943 "rpc worker-1494" 0x00007f2f03e1ead3 in ?? ()
112 LWP 14944 "rpc worker-1494" 0x00007f2f03e1ead3 in ?? ()
113 LWP 14945 "rpc worker-1494" 0x00007f2f03e1ead3 in ?? ()
114 LWP 14946 "rpc worker-1494" 0x00007f2f03e1ead3 in ?? ()
115 LWP 14947 "rpc worker-1494" 0x00007f2f03e1ead3 in ?? ()
116 LWP 14948 "rpc worker-1494" 0x00007f2f03e1ead3 in ?? ()
117 LWP 14949 "diag-logger-149" 0x00007f2f03e1efb9 in ?? ()
118 LWP 14950 "result-tracker-" 0x00007f2f03e1efb9 in ?? ()
119 LWP 14951 "excess-log-dele" 0x00007f2f03e1efb9 in ?? ()
120 LWP 14952 "acceptor-14952" 0x00007f2efcb530c7 in ?? ()
121 LWP 14953 "heartbeat-14953" 0x00007f2f03e1efb9 in ?? ()
122 LWP 14956 "maintenance_sch" 0x00007f2f03e1efb9 in ?? ()
Thread 122 (LWP 14956):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x00007f2eb865f6e0 in ?? ()
#2 0x0000000000000326 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f2eb865f730 in ?? ()
#6 0x000000000000064c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 14953):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x8272756aa36a08ac in ?? ()
#2 0x00000000000000c7 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007f2eb96865f0 in ?? ()
#6 0x000000000000018e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 14952):
#0 0x00007f2efcb530c7 in ?? ()
#1 0x00006140000210a8 in ?? ()
#2 0x00007f2eb9eadd90 in ?? ()
#3 0x00007f2eb9eaddc0 in ?? ()
#4 0x00007f2eb9eadec0 in ?? ()
#5 0x00007f2eb9eaddb0 in ?? ()
#6 0x00007f2eb9eade20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b082b in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 14951):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x00007f2eba6c6f40 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffc37024c70 in ?? ()
#5 0x00007f2eba6c6f30 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 14950):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x00007f2ebaee0100 in ?? ()
#2 0x00000000000000ca in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f2ebaee00f0 in ?? ()
#6 0x0000000000000194 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 14949):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 14948):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b6848 in ?? ()
#4 0x00007f2ebbf0fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2ebbf0feb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 14947):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x000000000000000c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f2ebc727e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2ebc727eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 14946):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 14945):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 14944):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 14943):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 14942):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 14941):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 14940):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 14939):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 14938):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 14937):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 14936):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 14935):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 14934):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 14933):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 14932):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 14931):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 14930):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 14929):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 14928):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 14927):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 14926):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 14925):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 14924):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 14923):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 14922):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 14921):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 14920):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 14918):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 14917):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 14916):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 14914):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 14913):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 14912):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 14911):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 14910):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 14909):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 14908):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 14907):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 14906):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x000000000000090d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f2ed02e0e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2ed02e0eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f2f03e1e770 in ?? ()
#10 0x00007f2ed02e0eb0 in ?? ()
#11 0x00000fe5da05c1d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 14905):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000576 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007f2ed0af8e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2ed0af8eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 14904):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000005 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a008c in ?? ()
#4 0x00007f2ed1310e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2ed1310eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a0040 in ?? ()
#9 0x00007f2f03e1e770 in ?? ()
#10 0x00007f2ed1310eb0 in ?? ()
#11 0x00000fe5da2621d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 73 (LWP 14903):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 14902):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 14901):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 14900):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 14899):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 14898):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 14897):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 14896):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 14895):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 14894):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 14893):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 14892):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 14891):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 14890):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 14889):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 14888):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 14887):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 14886):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000007 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f2eda4c1e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2eda4c1eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f2f03e1e770 in ?? ()
#10 0x00007f2eda4c1eb0 in ?? ()
#11 0x00000fe5db4983d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 14885):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 14884):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 14883):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 14882):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 14881):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 14880):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 14879):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 14878):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 14877):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 14876):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 14875):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 14874):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 14873):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 14872):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 14871):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 14870):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 14869):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 14868):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 14867):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 14866):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000007 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f2ee3e8ae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2ee3e8aeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f2f03e1e770 in ?? ()
#10 0x00007f2ee3e8aeb0 in ?? ()
#11 0x00000fe5dc7d15d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 14865):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 14864):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 14863):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 14862):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 14861):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 14860):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 14859):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 14858):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 14857):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 14856):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 14855):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 14854):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 14853):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 14852):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 14851):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 14850):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 14849):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 14848):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 14847):
#0 0x00007f2f03e1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 14846):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x00007f2eee09fcc0 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a560 in ?? ()
#5 0x00007f2eee09fcb0 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 14845):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 14843):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x00007f2eef927240 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d1a8 in ?? ()
#5 0x00007f2eef927230 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 14842):
#0 0x00000000009f49c8 in __sanitizer::theDepot ()
#1 0x00007f2ef014b100 in ?? ()
#2 0x00000fe5de029620 in ?? ()
#3 0x00007f2ef014b1e0 in ?? ()
#4 0x00000000009f479c in __sanitizer::theDepot ()
#5 0x0000000041b58ab3 in ?? ()
W20250625 03:47:05.372437 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 276: this message will repeat every 5th retry.
I20250625 03:47:05.980026 16760 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 10 similar messages]
#6 0x0000000000481fa4 in std::thread::~thread (this=0x61100018a2b0) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:133
#7 0x00000000004835ac in kudu::itest::RollingRestartITest::RollingRestartTServers(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)::{lambda()#2}::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/maintenance_mode-itest.cc:716
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
Thread 12 (LWP 14841):
#0 0x00007f2efcb51a47 in ?? ()
#1 0x00007f2ef0976320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f2ef0976310 in ?? ()
#4 0x00007f2ef0976520 in ?? ()
#5 0x00007f2ef0976360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f2ef09763e0 in ?? ()
#8 0x00007f2eff1c425d in ?? ()
#9 0x3fb979afd0d08000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000021800000 in ?? ()
#14 0x00007f2e00000000 in ?? ()
#15 0x41da16db2f4cc7aa in ?? ()
#16 0x00000fe65e126c7c in ?? ()
#17 0x00007f2ef09763c0 in ?? ()
#18 0x00007f2eff1c8ba3 in ?? ()
#19 0x00007f2ef0976390 in ?? ()
#20 0x3fb979afd0d08000 in ?? ()
#21 0x00000000f09763e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb979afd0d08000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 14840):
#0 0x00007f2efcb51a47 in ?? ()
#1 0x00007f2ef118d320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f2ef118d310 in ?? ()
#4 0x00007f2ef118d520 in ?? ()
#5 0x00007f2ef118d360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f2ef118d3e0 in ?? ()
#8 0x00007f2eff1c425d in ?? ()
#9 0x3fb95e7a2f1cc000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x000000005c400000 in ?? ()
#14 0x00007f2e00000000 in ?? ()
#15 0x41da16db2f4cc7ae in ?? ()
#16 0x00000fe65e229a7c in ?? ()
#17 0x00007f2ef118d3c0 in ?? ()
#18 0x00007f2eff1c8ba3 in ?? ()
#19 0x00007f2ef118d390 in ?? ()
#20 0x3fb95e7a2f1cc000 in ?? ()
#21 0x00000000f118d3e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021098 in ?? ()
#24 0x3fb95e7a2f1cc000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 14839):
#0 0x00007f2efcb51a47 in ?? ()
#1 0x00007f2ef19a4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f2ef19a4310 in ?? ()
#4 0x00007f2ef19a4520 in ?? ()
#5 0x00007f2ef19a4360 in ?? ()
#6 0x0000614000020e98 in ?? ()
#7 0x00007f2ef19a43e0 in ?? ()
#8 0x00007f2eff1c425d in ?? ()
#9 0x3fb96936ad8a4000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x0000000003800000 in ?? ()
#14 0x00007f2e00000000 in ?? ()
#15 0x41da16db2f4cc7ac in ?? ()
#16 0x00000fe65e32c87c in ?? ()
#17 0x00007f2ef19a43c0 in ?? ()
#18 0x00007f2eff1c8ba3 in ?? ()
#19 0x00007f2ef19a4390 in ?? ()
#20 0x3fb96936ad8a4000 in ?? ()
#21 0x00000000f19a43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000020e98 in ?? ()
#24 0x3fb96936ad8a4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 14838):
#0 0x00007f2efcb51a47 in ?? ()
#1 0x00007f2ef39ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f2ef39ad310 in ?? ()
#4 0x00007f2ef39ad520 in ?? ()
#5 0x00007f2ef39ad360 in ?? ()
#6 0x0000614000020c98 in ?? ()
#7 0x00007f2ef39ad3e0 in ?? ()
#8 0x00007f2eff1c425d in ?? ()
#9 0x3fadc51c0eb58000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000f8640000 in ?? ()
#14 0x00007f2e00000000 in ?? ()
#15 0x41da16db2f4cc7ac in ?? ()
#16 0x00000fe65e72da7c in ?? ()
#17 0x00007f2ef39ad3c0 in ?? ()
#18 0x00007f2eff1c8ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 14834):
#0 0x00007f2efcb44cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f2ef53b77c8 in ?? ()
#3 0x000060200001dc90 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a8471 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 14833):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x00007f2ef4bb60c0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007f2ef4bb60b0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 14831):
#0 0x00007f2f03e229e2 in ?? ()
#1 0x00007f2ef43b3ba0 in ?? ()
#2 0x00007f2ef43b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f2ef43b3bf0 in ?? ()
#6 0x000000000177e760 in ?? ()
#7 0x00007f2ef43b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f2ef43b3c40 in ?? ()
#10 0x00000000008a2967 in __sanitizer::theDepot ()
#11 0x00007f2f09735dcc in ?? ()
#12 0x00007f2f097257c9 in ?? ()
#13 0x00007f2f09729db6 in ?? ()
#14 0x00007f2f0972e7f0 in ?? ()
#15 0x00007f2f0972e619 in ?? ()
#16 0x0000000000a751bd in __sanitizer::theDepot ()
#17 0x00007f2f009f9f1e in ?? ()
#18 0x00007f2f03e186db in ?? ()
#19 0x00000fe65e86e684 in ?? ()
#20 0x00007f2ef43b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 14825):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x00007f2ef63b8c80 in ?? ()
#2 0x00000000000003f3 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007f2ef63b8c70 in ?? ()
#6 0x00000000000007e6 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 14824):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 14823):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 14822):
#0 0x00007f2f03e1efb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f2ef7bbc450 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 14819):
#0 0x00007f2f03e22d50 in ?? ()
#1 0x0000000000000000 in ?? ()
I20250625 03:47:06.746824 16770 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
W20250625 03:47:06.763185 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 281: this message will repeat every 5th retry.
I20250625 03:47:06.755352 16770 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:47:06.786484 14301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" is_pre_election: true
I20250625 03:47:06.793473 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:47:06.799196 16770 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:47:06.801738 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:47:06.804024 14841 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d275425c3ef443f88abb12d24f4c2f2f; no voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:47:06.805661 16770 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
************************* END STACKS ***************************
I20250625 03:47:06.852854 12990 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 4 with UUID 506230f3ca974bf796583a6be750e38f and pid 15840
W20250625 03:47:07.155541 14840 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.175.133:36385 timed out after 0.364s (SENT)
I20250625 03:47:07.193547 15931 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
I20250625 03:47:07.193249 15932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:47:07.200263 14840 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
W20250625 03:47:08.444067 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 286: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
I20250625 03:47:09.101015 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (5.43M/6.60M this segment, stats: ops{read=3230 overwritten=0 applied=3227 ignored=0} inserts{seen=161250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250625 03:47:09.171448 16769 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 9 similar messages]
W20250625 03:47:09.512787 15869 log.cc:927] Time spent T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Append to log took a long time: real 0.073s user 0.004s sys 0.000s
W20250625 03:47:10.064405 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 291: this message will repeat every 5th retry.
W20250625 03:47:11.176095 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 296: this message will repeat every 5th retry.
I20250625 03:47:12.207433 16779 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 10 similar messages]
W20250625 03:47:12.928285 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 301: this message will repeat every 5th retry.
[New LWP 15847]
[New LWP 15848]
[New LWP 15849]
[New LWP 15850]
[New LWP 15857]
[New LWP 15858]
[New LWP 15859]
[New LWP 15863]
[New LWP 15864]
[New LWP 15865]
[New LWP 15866]
[New LWP 15867]
[New LWP 15868]
[New LWP 15869]
[New LWP 15871]
[New LWP 15872]
[New LWP 15873]
[New LWP 15874]
[New LWP 15875]
[New LWP 15876]
[New LWP 15877]
[New LWP 15878]
[New LWP 15879]
[New LWP 15880]
[New LWP 15881]
[New LWP 15882]
[New LWP 15883]
[New LWP 15884]
[New LWP 15885]
[New LWP 15886]
[New LWP 15887]
[New LWP 15888]
[New LWP 15889]
[New LWP 15890]
[New LWP 15891]
[New LWP 15892]
[New LWP 15893]
[New LWP 15894]
[New LWP 15895]
[New LWP 15896]
[New LWP 15897]
[New LWP 15898]
[New LWP 15899]
[New LWP 15900]
[New LWP 15901]
[New LWP 15902]
[New LWP 15903]
[New LWP 15904]
[New LWP 15905]
[New LWP 15906]
[New LWP 15907]
[New LWP 15908]
[New LWP 15910]
[New LWP 15911]
[New LWP 15912]
[New LWP 15913]
[New LWP 15914]
[New LWP 15915]
[New LWP 15916]
[New LWP 15917]
[New LWP 15918]
[New LWP 15919]
[New LWP 15920]
[New LWP 15921]
[New LWP 15922]
[New LWP 15923]
[New LWP 15924]
[New LWP 15925]
[New LWP 15926]
[New LWP 15927]
[New LWP 15928]
[New LWP 15929]
[New LWP 15930]
[New LWP 15931]
[New LWP 15932]
[New LWP 15933]
[New LWP 15934]
[New LWP 15935]
[New LWP 15936]
[New LWP 15937]
[New LWP 15938]
[New LWP 15939]
[New LWP 15940]
[New LWP 15941]
[New LWP 15942]
[New LWP 15943]
[New LWP 15944]
[New LWP 15945]
[New LWP 15946]
[New LWP 15947]
[New LWP 15948]
[New LWP 15949]
[New LWP 15950]
[New LWP 15952]
[New LWP 15953]
[New LWP 15954]
[New LWP 15955]
[New LWP 15956]
[New LWP 15957]
[New LWP 15958]
[New LWP 15959]
[New LWP 15960]
[New LWP 15961]
[New LWP 15962]
[New LWP 15963]
[New LWP 15964]
[New LWP 15965]
[New LWP 15967]
[New LWP 15968]
[New LWP 15970]
[New LWP 15972]
[New LWP 15973]
[New LWP 15974]
[New LWP 15976]
[New LWP 15977]
[New LWP 15979]
[New LWP 15981]
[New LWP 15982]
[New LWP 15984]
[New LWP 15986]
[New LWP 15988]
[New LWP 15989]
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
0x00007f1507c22d50 in ?? ()
Id Target Id Frame
* 1 LWP 15840 "kudu" 0x00007f1507c22d50 in ?? ()
2 LWP 15847 "kudu" 0x00007f1507c1efb9 in ?? ()
3 LWP 15848 "kudu" 0x00007f1507c1efb9 in ?? ()
4 LWP 15849 "kudu" 0x00007f1507c1efb9 in ?? ()
5 LWP 15850 "kernel-watcher-" 0x00007f1507c1efb9 in ?? ()
6 LWP 15857 "ntp client-1585" 0x00007f1507c229e2 in ?? ()
7 LWP 15858 "file cache-evic" 0x00007f1507c1efb9 in ?? ()
8 LWP 15859 "sq_acceptor" 0x00007f1500944cb9 in ?? ()
9 LWP 15863 "rpc reactor-158" 0x00007f1500951a47 in ?? ()
10 LWP 15864 "rpc reactor-158" 0x00007f1500951a47 in ?? ()
11 LWP 15865 "rpc reactor-158" 0x00007f1500951a47 in ?? ()
12 LWP 15866 "rpc reactor-158" 0x00007f1500951a47 in ?? ()
13 LWP 15867 "MaintenanceMgr " 0x00007f1507c1ead3 in ?? ()
14 LWP 15868 "txn-status-mana" 0x00007f1507c1efb9 in ?? ()
15 LWP 15869 "tablet-open [wo" 0x000000000095d9f0 in __sanitizer::theDepot ()
16 LWP 15871 "collect_and_rem" 0x00007f1507c1efb9 in ?? ()
17 LWP 15872 "tc-session-exp-" 0x00007f1507c1efb9 in ?? ()
18 LWP 15873 "rpc worker-1587" 0x00007f1507c1ead3 in ?? ()
19 LWP 15874 "rpc worker-1587" 0x00007f1507c1ead3 in ?? ()
20 LWP 15875 "rpc worker-1587" 0x00007f1507c1ead3 in ?? ()
21 LWP 15876 "rpc worker-1587" 0x00007f1507c1ead3 in ?? ()
22 LWP 15877 "rpc worker-1587" 0x00007f1507c1ead3 in ?? ()
23 LWP 15878 "rpc worker-1587" 0x00007f1507c1ead3 in ?? ()
24 LWP 15879 "rpc worker-1587" 0x00007f1507c1ead3 in ?? ()
25 LWP 15880 "rpc worker-1588" 0x00007f1507c1ead3 in ?? ()
26 LWP 15881 "rpc worker-1588" 0x00007f1507c1ead3 in ?? ()
27 LWP 15882 "rpc worker-1588" 0x00007f1507c1ead3 in ?? ()
28 LWP 15883 "rpc worker-1588" 0x00007f1507c1ead3 in ?? ()
29 LWP 15884 "rpc worker-1588" 0x00007f1507c1ead3 in ?? ()
30 LWP 15885 "rpc worker-1588" 0x00007f1507c1ead3 in ?? ()
31 LWP 15886 "rpc worker-1588" 0x00007f1507c1ead3 in ?? ()
32 LWP 15887 "rpc worker-1588" 0x00007f1507c1ead3 in ?? ()
33 LWP 15888 "rpc worker-1588" 0x00007f1507c1ead3 in ?? ()
34 LWP 15889 "rpc worker-1588" 0x00007f1507c1ead3 in ?? ()
35 LWP 15890 "rpc worker-1589" 0x00007f1507c1ead3 in ?? ()
36 LWP 15891 "rpc worker-1589" 0x00007f1507c1ead3 in ?? ()
37 LWP 15892 "rpc worker-1589" 0x00007f1507c1ead3 in ?? ()
38 LWP 15893 "rpc worker-1589" 0x00007f1507c1ead3 in ?? ()
39 LWP 15894 "rpc worker-1589" 0x00007f1507c1ead3 in ?? ()
40 LWP 15895 "rpc worker-1589" 0x00007f1507c1ead3 in ?? ()
41 LWP 15896 "rpc worker-1589" 0x00007f1507c1ead3 in ?? ()
42 LWP 15897 "rpc worker-1589" 0x00007f1507c1ead3 in ?? ()
43 LWP 15898 "rpc worker-1589" 0x00007f1507c1ead3 in ?? ()
44 LWP 15899 "rpc worker-1589" 0x00007f1507c1ead3 in ?? ()
45 LWP 15900 "rpc worker-1590" 0x00007f1507c1ead3 in ?? ()
46 LWP 15901 "rpc worker-1590" 0x00007f1507c1ead3 in ?? ()
47 LWP 15902 "rpc worker-1590" 0x00007f1507c1ead3 in ?? ()
48 LWP 15903 "rpc worker-1590" 0x00007f1507c1ead3 in ?? ()
49 LWP 15904 "rpc worker-1590" 0x00007f1507c1ead3 in ?? ()
50 LWP 15905 "rpc worker-1590" 0x00007f1507c1ead3 in ?? ()
51 LWP 15906 "rpc worker-1590" 0x00007f1507c1ead3 in ?? ()
52 LWP 15907 "rpc worker-1590" 0x00007f1507c1ead3 in ?? ()
53 LWP 15908 "rpc worker-1590" 0x00007f1507c1ead3 in ?? ()
54 LWP 15910 "rpc worker-1591" 0x00007f1507c1ead3 in ?? ()
55 LWP 15911 "rpc worker-1591" 0x00007f1507c1ead3 in ?? ()
56 LWP 15912 "rpc worker-1591" 0x00007f1507c1ead3 in ?? ()
57 LWP 15913 "rpc worker-1591" 0x00007f1507c1ead3 in ?? ()
58 LWP 15914 "rpc worker-1591" 0x00007f1507c1ead3 in ?? ()
59 LWP 15915 "rpc worker-1591" 0x00007f1507c1ead3 in ?? ()
60 LWP 15916 "rpc worker-1591" 0x00007f1507c1ead3 in ?? ()
61 LWP 15917 "rpc worker-1591" 0x00007f1507c1ead3 in ?? ()
62 LWP 15918 "rpc worker-1591" 0x00007f1507c1ead3 in ?? ()
63 LWP 15919 "rpc worker-1591" 0x00007f1507c1ead3 in ?? ()
64 LWP 15920 "rpc worker-1592" 0x00007f1507c1ead3 in ?? ()
65 LWP 15921 "rpc worker-1592" 0x00007f1507c1ead3 in ?? ()
66 LWP 15922 "rpc worker-1592" 0x00007f1507c1ead3 in ?? ()
67 LWP 15923 "rpc worker-1592" 0x00007f1507c1ead3 in ?? ()
68 LWP 15924 "rpc worker-1592" 0x00007f1507c1ead3 in ?? ()
69 LWP 15925 "rpc worker-1592" 0x00007f1507c1ead3 in ?? ()
70 LWP 15926 "rpc worker-1592" 0x00007f1507c1ead3 in ?? ()
71 LWP 15927 "rpc worker-1592" 0x00007f1507c1ead3 in ?? ()
72 LWP 15928 "rpc worker-1592" 0x00007f1507c1ead3 in ?? ()
73 LWP 15929 "rpc worker-1592" 0x00007f1507c1ead3 in ?? ()
74 LWP 15930 "rpc worker-1593" 0x00007f1507c1ead3 in ?? ()
75 LWP 15931 "rpc worker-1593" 0x00007f1507c1ead3 in ?? ()
76 LWP 15932 "rpc worker-1593" 0x00007f1507c1ead3 in ?? ()
77 LWP 15933 "rpc worker-1593" 0x00007f1507c1ead3 in ?? ()
78 LWP 15934 "rpc worker-1593" 0x00007f1507c1ead3 in ?? ()
79 LWP 15935 "rpc worker-1593" 0x00007f1507c1ead3 in ?? ()
80 LWP 15936 "rpc worker-1593" 0x00007f1507c1ead3 in ?? ()
81 LWP 15937 "rpc worker-1593" 0x00007f1507c1ead3 in ?? ()
82 LWP 15938 "rpc worker-1593" 0x00007f1507c1ead3 in ?? ()
83 LWP 15939 "rpc worker-1593" 0x00007f1507c1ead3 in ?? ()
84 LWP 15940 "rpc worker-1594" 0x00007f1507c1ead3 in ?? ()
85 LWP 15941 "rpc worker-1594" 0x00007f1507c1ead3 in ?? ()
86 LWP 15942 "rpc worker-1594" 0x00007f1507c1ead3 in ?? ()
87 LWP 15943 "rpc worker-1594" 0x00007f1507c1ead3 in ?? ()
88 LWP 15944 "rpc worker-1594" 0x00007f1507c1ead3 in ?? ()
89 LWP 15945 "rpc worker-1594" 0x00007f1507c1ead3 in ?? ()
90 LWP 15946 "rpc worker-1594" 0x00007f1507c1ead3 in ?? ()
91 LWP 15947 "rpc worker-1594" 0x00007f1507c1ead3 in ?? ()
92 LWP 15948 "rpc worker-1594" 0x00007f1507c1ead3 in ?? ()
93 LWP 15949 "rpc worker-1594" 0x00007f1507c1ead3 in ?? ()
94 LWP 15950 "rpc worker-1595" 0x00007f1507c1ead3 in ?? ()
95 LWP 15952 "rpc worker-1595" 0x00007f1507c1ead3 in ?? ()
96 LWP 15953 "rpc worker-1595" 0x00007f1507c1ead3 in ?? ()
97 LWP 15954 "rpc worker-1595" 0x00007f1507c1ead3 in ?? ()
98 LWP 15955 "rpc worker-1595" 0x00007f1507c1ead3 in ?? ()
99 LWP 15956 "rpc worker-1595" 0x00007f1507c1ead3 in ?? ()
100 LWP 15957 "rpc worker-1595" 0x00007f1507c1ead3 in ?? ()
101 LWP 15958 "rpc worker-1595" 0x00007f1507c1ead3 in ?? ()
102 LWP 15959 "rpc worker-1595" 0x00007f1507c1ead3 in ?? ()
103 LWP 15960 "rpc worker-1596" 0x00007f1507c1ead3 in ?? ()
104 LWP 15961 "rpc worker-1596" 0x00007f1507c1ead3 in ?? ()
105 LWP 15962 "rpc worker-1596" 0x00007f1507c1ead3 in ?? ()
106 LWP 15963 "rpc worker-1596" 0x00007f1507c1ead3 in ?? ()
107 LWP 15964 "rpc worker-1596" 0x00007f1507c1ead3 in ?? ()
108 LWP 15965 "rpc worker-1596" 0x00007f1507c1ead3 in ?? ()
109 LWP 15967 "rpc worker-1596" 0x00007f1507c1ead3 in ?? ()
110 LWP 15968 "rpc worker-1596" 0x00007f1507c1ead3 in ?? ()
111 LWP 15970 "rpc worker-1597" 0x00007f1507c1ead3 in ?? ()
112 LWP 15972 "rpc worker-1597" 0x00007f1507c1ead3 in ?? ()
113 LWP 15973 "rpc worker-1597" 0x00007f1507c1ead3 in ?? ()
114 LWP 15974 "rpc worker-1597" 0x00007f1507c1ead3 in ?? ()
115 LWP 15976 "rpc worker-1597" 0x00007f1507c1ead3 in ?? ()
116 LWP 15977 "rpc worker-1597" 0x00007f1507c1ead3 in ?? ()
117 LWP 15979 "rpc worker-1597" 0x00007f1507c1ead3 in ?? ()
118 LWP 15981 "diag-logger-159" 0x00007f1507c1efb9 in ?? ()
119 LWP 15982 "result-tracker-" 0x00007f1507c1efb9 in ?? ()
120 LWP 15984 "excess-log-dele" 0x00007f1507c1efb9 in ?? ()
121 LWP 15986 "acceptor-15986" 0x00007f15009530c7 in ?? ()
122 LWP 15988 "heartbeat-15988" 0x00007f1507c1efb9 in ?? ()
123 LWP 15989 "maintenance_sch" 0x00007f1507c1efb9 in ?? ()
Thread 123 (LWP 15989):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x00007f14bcc6e6e0 in ?? ()
#2 0x00000000000001f8 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f14bcc6e730 in ?? ()
#6 0x00000000000003f0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 15988):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x8272756aa36a08ac in ?? ()
#2 0x000000000000007c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007f14bd4865f0 in ?? ()
#6 0x00000000000000f8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 15986):
#0 0x00007f15009530c7 in ?? ()
#1 0x00006140000212a8 in ?? ()
#2 0x00007f14bdcadd90 in ?? ()
#3 0x00007f14bdcaddc0 in ?? ()
#4 0x00007f14bdcadec0 in ?? ()
#5 0x00007f14bdcaddb0 in ?? ()
#6 0x00007f14bdcade20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b082b in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007f14bdcadf40 in ?? ()
#11 0x00007f14bdcad67c in ?? ()
#12 0x00000032bdcad5f0 in ?? ()
#13 0x00007f14bd4b1000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 120 (LWP 15984):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x00007f14be4c6f40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffeea6d5910 in ?? ()
#5 0x00007f14be4c6f30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 15982):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x00007f14bece0100 in ?? ()
#2 0x000000000000007e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f14bece00f0 in ?? ()
#6 0x00000000000000fc in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 15981):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 117 (LWP 15979):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000007 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b684c in ?? ()
#4 0x00007f14bfd0fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f14bfd0feb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b6800 in ?? ()
#9 0x00007f1507c1e770 in ?? ()
#10 0x00007f14bfd0feb0 in ?? ()
#11 0x00000fe297fa1fd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 116 (LWP 15977):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000005 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f14c0527e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f14c0527eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f1507c1e770 in ?? ()
#10 0x00007f14c0527eb0 in ?? ()
#11 0x00000fe2980a4fd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 15976):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 15974):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 15973):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 15972):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 15970):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 15968):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 15967):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 15965):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 15964):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 15963):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 15962):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 15961):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 15960):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 15959):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 15958):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 15957):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 15956):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 15955):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 15954):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 15953):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 15952):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 15950):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 15949):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 15948):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 15947):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 15946):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 15945):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 15944):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 15943):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 15942):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 15941):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 15940):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 15939):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 15938):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 15937):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 15936):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 15935):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 15934):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 15933):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x000000000000011b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f14d40e0e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f14d40e0eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f1507c1e770 in ?? ()
#10 0x00007f14d40e0eb0 in ?? ()
#11 0x00000fe29a81c1d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 76 (LWP 15932):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007f14d48f8e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f14d48f8eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 15931):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a008c in ?? ()
#4 0x00007f14d5110e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f14d5110eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a0040 in ?? ()
#9 0x00007f1507c1e770 in ?? ()
#10 0x00007f14d5110eb0 in ?? ()
#11 0x00000fe29aa221d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 15930):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 15929):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 15928):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 15927):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 15926):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 15925):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 15924):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 15923):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 15922):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 15921):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 15920):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 15919):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 15918):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 15917):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 15916):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 15915):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 15914):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 15913):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f14de2c1e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f14de2c1eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f1507c1e770 in ?? ()
#10 0x00007f14de2c1eb0 in ?? ()
#11 0x00000fe29bc583d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 15912):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 15911):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 15910):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 15908):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 15907):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 15906):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 15905):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 15904):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 15903):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 15902):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 15901):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 15900):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 15899):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 15898):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 15897):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 15896):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 15895):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 15894):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 15893):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 15892):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f14e84a2e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f14e84a2eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f1507c1e770 in ?? ()
#10 0x00007f14e84a2eb0 in ?? ()
#11 0x00000fe29d0945d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 15891):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 15890):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 15889):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 15888):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 15887):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 15886):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 15885):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 15884):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 15883):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 15882):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 15881):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 15880):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 15879):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 15878):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 15877):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 15876):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 15875):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 15874):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 15873):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 15872):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x00007f14f1dc5cc0 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a560 in ?? ()
#5 0x00007f14f1dc5cb0 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 15871):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 15 (LWP 15869):
#0 0x000000000095d9f0 in __sanitizer::theDepot ()
#1 0x00007f150464e429 in ?? ()
#2 0x0000000041b58ab3 in ?? ()
#3 0x00007f15041502a1 in ?? ()
#4 0x00007f150464e2f0 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 14 (LWP 15868):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x00007f14f35c8240 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d1a8 in ?? ()
#5 0x00007f14f35c8230 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 15867):
#0 0x00007f1507c1ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 15866):
#0 0x00007f1500951a47 in ?? ()
#1 0x00007f14f45cb320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f14f45cb310 in ?? ()
#4 0x00007f14f45cb520 in ?? ()
#5 0x00007f14f45cb360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f14f45cb3e0 in ?? ()
#8 0x00007f1502fc425d in ?? ()
#9 0x3fb96ac762a54000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000025000000 in ?? ()
#14 0x00007f1400000000 in ?? ()
#15 0x41da16db2f4cc7ae in ?? ()
#16 0x00000fe31e8b167c in ?? ()
#17 0x00007f14f45cb3c0 in ?? ()
#18 0x00007f1502fc8ba3 in ?? ()
#19 0x00007f14f45cb390 in ?? ()
#20 0x3fb96ac762a54000 in ?? ()
#21 0x00000000f45cb3e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb96ac762a54000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 15865):
#0 0x00007f1500951a47 in ?? ()
#1 0x00007f14f4dcc320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f14f4dcc310 in ?? ()
#4 0x00007f14f4dcc520 in ?? ()
#5 0x00007f14f4dcc360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f14f4dcc3e0 in ?? ()
#8 0x00007f1502fc425d in ?? ()
#9 0x3fa91b5421f00000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000a1d18000 in ?? ()
#14 0x00007f1400000000 in ?? ()
#15 0x41da16db2f4cc7ad in ?? ()
#16 0x00000fe31e9b187c in ?? ()
#17 0x00007f14f4dcc3c0 in ?? ()
#18 0x00007f1502fc8ba3 in ?? ()
#19 0x00007f14f4dcc390 in ?? ()
#20 0x3fa91b5421f00000 in ?? ()
#21 0x00000000f4dcc3e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021098 in ?? ()
#24 0x3fa91b5421f00000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 15864):
#0 0x00007f1500951a47 in ?? ()
#1 0x00007f14f55cd320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f14f55cd310 in ?? ()
#4 0x00007f14f55cd520 in ?? ()
#5 0x00007f14f55cd360 in ?? ()
#6 0x0000614000020e98 in ?? ()
#7 0x00007f14f55cd3e0 in ?? ()
#8 0x00007f1502fc425d in ?? ()
#9 0x3fb9887d518ec000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x0000000005000000 in ?? ()
#14 0x00007f1400000000 in ?? ()
#15 0x41da16db2f4cc7ad in ?? ()
#16 0x00000fe31eab1a7c in ?? ()
#17 0x00007f14f55cd3c0 in ?? ()
#18 0x00007f1502fc8ba3 in ?? ()
#19 0x00007f14f55cd390 in ?? ()
#20 0x3fb9887d518ec000 in ?? ()
#21 0x00000000f55cd3e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000020e98 in ?? ()
#24 0x3fb9887d518ec000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 15863):
#0 0x00007f1500951a47 in ?? ()
#1 0x00007f14f71b3320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f14f71b3310 in ?? ()
#4 0x00007f14f71b3520 in ?? ()
#5 0x00007f14f71b3360 in ?? ()
#6 0x0000614000020c98 in ?? ()
#7 0x00007f14f71b33e0 in ?? ()
#8 0x00007f1502fc425d in ?? ()
#9 0x3fb981cf73abc000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000f6c00000 in ?? ()
#14 0x00007f1400000000 in ?? ()
#15 0x41da16db2f4cc7aa in ?? ()
#16 0x00000fe31ee2e67c in ?? ()
#17 0x00007f14f71b33c0 in ?? ()
#18 0x00007f1502fc8ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 15859):
#0 0x00007f1500944cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f14f91b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a8471 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 15858):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x00007f14f89b60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007f14f89b60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 15857):
#0 0x00007f1507c229e2 in ?? ()
#1 0x00007f14f79b2ba0 in ?? ()
#2 0x00007f14f79b2c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f14f79b2bf0 in ?? ()
#6 0x000000000177e760 in ?? ()
#7 0x00007f14f79b2c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f14f79b2c40 in ?? ()
#10 0x00000000008a2967 in __sanitizer::theDepot ()
#11 0x00007f150d535dcc in ?? ()
#12 0x00007f150d5257c9 in ?? ()
#13 0x00007f150d529db6 in ?? ()
#14 0x00007f150d52e7f0 in ?? ()
#15 0x00007f150d52e619 in ?? ()
#16 0x0000000000a751bd in __sanitizer::theDepot ()
#17 0x00007f15047f9f1e in ?? ()
#18 0x00007f1507c186db in ?? ()
#19 0x00000fe31ef2e484 in ?? ()
#20 0x00007f14f79b2440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 15850):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x00007f14fa1b8c80 in ?? ()
#2 0x000000000000027f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007f14fa1b8c70 in ?? ()
#6 0x00000000000004fe in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 15849):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 15848):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 15847):
#0 0x00007f1507c1efb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f14fb9bc450 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 15840):
#0 0x00007f1507c22d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250625 03:47:13.930063 12990 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 5 with UUID c24f9318ff0c418292d80906d5f0fd95 and pid 15647
I20250625 03:47:14.123209 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (5.61M/6.60M this segment, stats: ops{read=3334 overwritten=0 applied=3334 ignored=0} inserts{seen=166600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250625 03:47:15.162128 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 306: this message will repeat every 5th retry.
I20250625 03:47:15.411424 16788 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 7 similar messages]
************************ BEGIN STACKS **************************
W20250625 03:47:16.483507 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 311: this message will repeat every 5th retry.
I20250625 03:47:17.176084 14234 maintenance_manager.cc:643] P 9cb1b4dfaea74515a9e5aab1fbc0870f: FlushMRSOp(9126b7f95c9d434cb308581a7ce5b908) complete. Timing: real 120.438s user 34.366s sys 2.535s Metrics: {"bytes_written":6161433,"cfile_init":1,"compiler_manager_pool.queue_time_us":19494,"compiler_manager_pool.run_cpu_time_us":2275911,"compiler_manager_pool.run_wall_time_us":15140580,"dirs.queue_time_us":27989,"dirs.run_cpu_time_us":594,"dirs.run_wall_time_us":594,"drs_written":1,"lbm_read_time_us":174,"lbm_reads_lt_1ms":4,"lbm_write_time_us":513662,"lbm_writes_1-10_ms":30,"lbm_writes_10-100_ms":15,"lbm_writes_lt_1ms":234,"peak_mem_usage":0,"rows_written":220400,"thread_start_us":90985,"threads_started":3,"wal-append.queue_time_us":7433}
I20250625 03:47:17.186743 14349 maintenance_manager.cc:419] P 9cb1b4dfaea74515a9e5aab1fbc0870f: Scheduling UndoDeltaBlockGCOp(9126b7f95c9d434cb308581a7ce5b908): 2529968 bytes on disk
I20250625 03:47:17.190621 14234 maintenance_manager.cc:643] P 9cb1b4dfaea74515a9e5aab1fbc0870f: UndoDeltaBlockGCOp(9126b7f95c9d434cb308581a7ce5b908) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":663,"lbm_reads_lt_1ms":4}
W20250625 03:47:17.701515 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 316: this message will repeat every 5th retry.
I20250625 03:47:18.525570 16796 consensus_queue.cc:786] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f [LEADER]: Peer 506230f3ca974bf796583a6be750e38f is lagging by at least 667 ops behind the committed index [suppressed 11 similar messages]
[New LWP 15671]
[New LWP 15672]
[New LWP 15673]
[New LWP 15677]
[New LWP 15685]
[New LWP 15688]
[New LWP 15690]
[New LWP 15695]
[New LWP 15696]
[New LWP 15697]
[New LWP 15698]
[New LWP 15699]
[New LWP 15700]
[New LWP 15702]
[New LWP 15703]
[New LWP 15705]
[New LWP 15706]
[New LWP 15707]
[New LWP 15708]
[New LWP 15710]
[New LWP 15711]
[New LWP 15712]
[New LWP 15713]
[New LWP 15714]
[New LWP 15715]
[New LWP 15716]
[New LWP 15717]
[New LWP 15719]
[New LWP 15720]
[New LWP 15721]
[New LWP 15722]
[New LWP 15723]
[New LWP 15724]
[New LWP 15725]
[New LWP 15726]
[New LWP 15727]
[New LWP 15728]
[New LWP 15729]
[New LWP 15730]
[New LWP 15732]
[New LWP 15733]
[New LWP 15734]
[New LWP 15735]
[New LWP 15736]
[New LWP 15737]
[New LWP 15738]
[New LWP 15739]
[New LWP 15740]
[New LWP 15742]
[New LWP 15743]
[New LWP 15744]
[New LWP 15745]
[New LWP 15746]
[New LWP 15747]
[New LWP 15748]
[New LWP 15752]
[New LWP 15753]
[New LWP 15755]
[New LWP 15756]
[New LWP 15757]
[New LWP 15758]
[New LWP 15759]
[New LWP 15760]
[New LWP 15761]
[New LWP 15763]
[New LWP 15765]
[New LWP 15766]
[New LWP 15767]
[New LWP 15768]
[New LWP 15769]
[New LWP 15770]
[New LWP 15771]
[New LWP 15772]
[New LWP 15774]
[New LWP 15775]
[New LWP 15776]
[New LWP 15778]
[New LWP 15779]
[New LWP 15781]
[New LWP 15782]
[New LWP 15783]
[New LWP 15784]
[New LWP 15785]
[New LWP 15787]
[New LWP 15788]
[New LWP 15789]
[New LWP 15790]
[New LWP 15791]
[New LWP 15792]
[New LWP 15793]
[New LWP 15794]
[New LWP 15795]
[New LWP 15796]
[New LWP 15797]
[New LWP 15798]
[New LWP 15799]
[New LWP 15800]
[New LWP 15801]
[New LWP 15802]
[New LWP 15803]
[New LWP 15804]
[New LWP 15805]
[New LWP 15806]
[New LWP 15807]
[New LWP 15809]
[New LWP 15811]
[New LWP 15813]
[New LWP 15814]
[New LWP 15815]
[New LWP 15816]
[New LWP 15817]
[New LWP 15818]
[New LWP 15819]
[New LWP 15820]
[New LWP 15821]
[New LWP 15822]
[New LWP 15823]
[New LWP 15824]
[New LWP 15825]
[New LWP 15826]
[New LWP 15830]
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a6d
Cannot access memory at address 0x5f6362696c3a3a65
0x00007fed2fba7d50 in ?? ()
Id Target Id Frame
* 1 LWP 15647 "kudu" 0x00007fed2fba7d50 in ?? ()
W20250625 03:47:18.842497 14232 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P 9cb1b4dfaea74515a9e5aab1fbc0870f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 321: this message will repeat every 5th retry.
2 LWP 15671 "kudu" 0x00007fed2fba3fb9 in ?? ()
3 LWP 15672 "kudu" 0x00007fed2fba3fb9 in ?? ()
4 LWP 15673 "kudu" 0x00007fed2fba3fb9 in ?? ()
5 LWP 15677 "kernel-watcher-" 0x00007fed2fba3fb9 in ?? ()
6 LWP 15685 "ntp client-1568" 0x00007fed2fba79e2 in ?? ()
7 LWP 15688 "file cache-evic" 0x00007fed2fba3fb9 in ?? ()
8 LWP 15690 "sq_acceptor" 0x00007fed288c9cb9 in ?? ()
9 LWP 15695 "rpc reactor-156" 0x00007fed288d6a47 in ?? ()
10 LWP 15696 "rpc reactor-156" 0x00007fed288d6a47 in ?? ()
11 LWP 15697 "rpc reactor-156" 0x00007fed288d6a47 in ?? ()
12 LWP 15698 "rpc reactor-156" 0x00007fed288d6a47 in ?? ()
13 LWP 15699 "MaintenanceMgr " 0x00007fed2fba3ad3 in ?? ()
14 LWP 15700 "txn-status-mana" 0x00007fed2fba3fb9 in ?? ()
15 LWP 15702 "collect_and_rem" 0x00007fed2fba3fb9 in ?? ()
16 LWP 15703 "tc-session-exp-" 0x00007fed2fba3fb9 in ?? ()
17 LWP 15705 "rpc worker-1570" 0x00007fed2fba3ad3 in ?? ()
18 LWP 15706 "rpc worker-1570" 0x00007fed2fba3ad3 in ?? ()
19 LWP 15707 "rpc worker-1570" 0x00007fed2fba3ad3 in ?? ()
20 LWP 15708 "rpc worker-1570" 0x00007fed2fba3ad3 in ?? ()
21 LWP 15710 "rpc worker-1571" 0x00007fed2fba3ad3 in ?? ()
22 LWP 15711 "rpc worker-1571" 0x00007fed2fba3ad3 in ?? ()
23 LWP 15712 "rpc worker-1571" 0x00007fed2fba3ad3 in ?? ()
24 LWP 15713 "rpc worker-1571" 0x00007fed2fba3ad3 in ?? ()
25 LWP 15714 "rpc worker-1571" 0x00007fed2fba3ad3 in ?? ()
26 LWP 15715 "rpc worker-1571" 0x00007fed2fba3ad3 in ?? ()
27 LWP 15716 "rpc worker-1571" 0x00007fed2fba3ad3 in ?? ()
28 LWP 15717 "rpc worker-1571" 0x00007fed2fba3ad3 in ?? ()
29 LWP 15719 "rpc worker-1571" 0x00007fed2fba3ad3 in ?? ()
30 LWP 15720 "rpc worker-1572" 0x00007fed2fba3ad3 in ?? ()
31 LWP 15721 "rpc worker-1572" 0x00007fed2fba3ad3 in ?? ()
32 LWP 15722 "rpc worker-1572" 0x00007fed2fba3ad3 in ?? ()
33 LWP 15723 "rpc worker-1572" 0x00007fed2fba3ad3 in ?? ()
34 LWP 15724 "rpc worker-1572" 0x00007fed2fba3ad3 in ?? ()
35 LWP 15725 "rpc worker-1572" 0x00007fed2fba3ad3 in ?? ()
36 LWP 15726 "rpc worker-1572" 0x00007fed2fba3ad3 in ?? ()
37 LWP 15727 "rpc worker-1572" 0x00007fed2fba3ad3 in ?? ()
38 LWP 15728 "rpc worker-1572" 0x00007fed2fba3ad3 in ?? ()
39 LWP 15729 "rpc worker-1572" 0x00007fed2fba3ad3 in ?? ()
40 LWP 15730 "rpc worker-1573" 0x00007fed2fba3ad3 in ?? ()
41 LWP 15732 "rpc worker-1573" 0x00007fed2fba3ad3 in ?? ()
42 LWP 15733 "rpc worker-1573" 0x00007fed2fba3ad3 in ?? ()
43 LWP 15734 "rpc worker-1573" 0x00007fed2fba3ad3 in ?? ()
44 LWP 15735 "rpc worker-1573" 0x00007fed2fba3ad3 in ?? ()
45 LWP 15736 "rpc worker-1573" 0x00007fed2fba3ad3 in ?? ()
46 LWP 15737 "rpc worker-1573" 0x00007fed2fba3ad3 in ?? ()
47 LWP 15738 "rpc worker-1573" 0x00007fed2fba3ad3 in ?? ()
48 LWP 15739 "rpc worker-1573" 0x00007fed2fba3ad3 in ?? ()
49 LWP 15740 "rpc worker-1574" 0x00007fed2fba3ad3 in ?? ()
50 LWP 15742 "rpc worker-1574" 0x00007fed2fba3ad3 in ?? ()
51 LWP 15743 "rpc worker-1574" 0x00007fed2fba3ad3 in ?? ()
52 LWP 15744 "rpc worker-1574" 0x00007fed2fba3ad3 in ?? ()
53 LWP 15745 "rpc worker-1574" 0x00007fed2fba3ad3 in ?? ()
54 LWP 15746 "rpc worker-1574" 0x00007fed2fba3ad3 in ?? ()
55 LWP 15747 "rpc worker-1574" 0x00007fed2fba3ad3 in ?? ()
56 LWP 15748 "rpc worker-1574" 0x00007fed2fba3ad3 in ?? ()
57 LWP 15752 "rpc worker-1575" 0x00007fed2fba3ad3 in ?? ()
58 LWP 15753 "rpc worker-1575" 0x00007fed2fba3ad3 in ?? ()
59 LWP 15755 "rpc worker-1575" 0x00007fed2fba3ad3 in ?? ()
60 LWP 15756 "rpc worker-1575" 0x00007fed2fba3ad3 in ?? ()
61 LWP 15757 "rpc worker-1575" 0x00007fed2fba3ad3 in ?? ()
62 LWP 15758 "rpc worker-1575" 0x00007fed2fba3ad3 in ?? ()
63 LWP 15759 "rpc worker-1575" 0x00007fed2fba3ad3 in ?? ()
64 LWP 15760 "rpc worker-1576" 0x00007fed2fba3ad3 in ?? ()
65 LWP 15761 "rpc worker-1576" 0x00007fed2fba3ad3 in ?? ()
66 LWP 15763 "rpc worker-1576" 0x00007fed2fba3ad3 in ?? ()
67 LWP 15765 "rpc worker-1576" 0x00007fed2fba3ad3 in ?? ()
68 LWP 15766 "rpc worker-1576" 0x00007fed2fba3ad3 in ?? ()
69 LWP 15767 "rpc worker-1576" 0x00007fed2fba3ad3 in ?? ()
70 LWP 15768 "rpc worker-1576" 0x00007fed2fba3ad3 in ?? ()
71 LWP 15769 "rpc worker-1576" 0x00007fed2fba3ad3 in ?? ()
72 LWP 15770 "rpc worker-1577" 0x00007fed2fba3ad3 in ?? ()
73 LWP 15771 "rpc worker-1577" 0x00007fed2fba3ad3 in ?? ()
74 LWP 15772 "rpc worker-1577" 0x00007fed2fba3ad3 in ?? ()
75 LWP 15774 "rpc worker-1577" 0x00007fed2fba3ad3 in ?? ()
76 LWP 15775 "rpc worker-1577" 0x00007fed2fba3ad3 in ?? ()
77 LWP 15776 "rpc worker-1577" 0x00007fed2fba3ad3 in ?? ()
78 LWP 15778 "rpc worker-1577" 0x00007fed2fba3ad3 in ?? ()
79 LWP 15779 "rpc worker-1577" 0x00007fed2fba3ad3 in ?? ()
80 LWP 15781 "rpc worker-1578" 0x00007fed2fba3ad3 in ?? ()
81 LWP 15782 "rpc worker-1578" 0x00007fed2fba3ad3 in ?? ()
82 LWP 15783 "rpc worker-1578" 0x00007fed2fba3ad3 in ?? ()
83 LWP 15784 "rpc worker-1578" 0x00007fed2fba3ad3 in ?? ()
84 LWP 15785 "rpc worker-1578" 0x00007fed2fba3ad3 in ?? ()
85 LWP 15787 "rpc worker-1578" 0x00007fed2fba3ad3 in ?? ()
86 LWP 15788 "rpc worker-1578" 0x00007fed2fba3ad3 in ?? ()
87 LWP 15789 "rpc worker-1578" 0x00007fed2fba3ad3 in ?? ()
88 LWP 15790 "rpc worker-1579" 0x00007fed2fba3ad3 in ?? ()
89 LWP 15791 "rpc worker-1579" 0x00007fed2fba3ad3 in ?? ()
90 LWP 15792 "rpc worker-1579" 0x00007fed2fba3ad3 in ?? ()
91 LWP 15793 "rpc worker-1579" 0x00007fed2fba3ad3 in ?? ()
92 LWP 15794 "rpc worker-1579" 0x00007fed2fba3ad3 in ?? ()
93 LWP 15795 "rpc worker-1579" 0x00007fed2fba3ad3 in ?? ()
94 LWP 15796 "rpc worker-1579" 0x00007fed2fba3ad3 in ?? ()
95 LWP 15797 "rpc worker-1579" 0x00007fed2fba3ad3 in ?? ()
96 LWP 15798 "rpc worker-1579" 0x00007fed2fba3ad3 in ?? ()
97 LWP 15799 "rpc worker-1579" 0x00007fed2fba3ad3 in ?? ()
98 LWP 15800 "rpc worker-1580" 0x00007fed2fba3ad3 in ?? ()
99 LWP 15801 "rpc worker-1580" 0x00007fed2fba3ad3 in ?? ()
100 LWP 15802 "rpc worker-1580" 0x00007fed2fba3ad3 in ?? ()
101 LWP 15803 "rpc worker-1580" 0x00007fed2fba3ad3 in ?? ()
102 LWP 15804 "rpc worker-1580" 0x00007fed2fba3ad3 in ?? ()
103 LWP 15805 "rpc worker-1580" 0x00007fed2fba3ad3 in ?? ()
104 LWP 15806 "rpc worker-1580" 0x00007fed2fba3ad3 in ?? ()
105 LWP 15807 "rpc worker-1580" 0x00007fed2fba3ad3 in ?? ()
106 LWP 15809 "rpc worker-1580" 0x00007fed2fba3ad3 in ?? ()
107 LWP 15811 "rpc worker-1581" 0x00007fed2fba3ad3 in ?? ()
108 LWP 15813 "rpc worker-1581" 0x00007fed2fba3ad3 in ?? ()
109 LWP 15814 "rpc worker-1581" 0x00007fed2fba3ad3 in ?? ()
110 LWP 15815 "rpc worker-1581" 0x00007fed2fba3ad3 in ?? ()
111 LWP 15816 "rpc worker-1581" 0x00007fed2fba3ad3 in ?? ()
112 LWP 15817 "rpc worker-1581" 0x00007fed2fba3ad3 in ?? ()
113 LWP 15818 "rpc worker-1581" 0x00007fed2fba3ad3 in ?? ()
114 LWP 15819 "rpc worker-1581" 0x00007fed2fba3ad3 in ?? ()
115 LWP 15820 "rpc worker-1582" 0x00007fed2fba3ad3 in ?? ()
116 LWP 15821 "rpc worker-1582" 0x00007fed2fba3ad3 in ?? ()
117 LWP 15822 "diag-logger-158" 0x00007fed2fba3fb9 in ?? ()
118 LWP 15823 "result-tracker-" 0x00007fed2fba3fb9 in ?? ()
119 LWP 15824 "excess-log-dele" 0x00007fed2fba3fb9 in ?? ()
120 LWP 15825 "acceptor-15825" 0x00007fed288d80c7 in ?? ()
121 LWP 15826 "heartbeat-15826" 0x00007fed2fba3fb9 in ?? ()
122 LWP 15830 "maintenance_sch" 0x00007fed2fba3fb9 in ?? ()
Thread 122 (LWP 15830):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x00007fece544b6e0 in ?? ()
#2 0x0000000000000225 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007fece544b730 in ?? ()
#6 0x000000000000044a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 15826):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x8272756aa36a08ac in ?? ()
#2 0x0000000000000087 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007fece5c635f0 in ?? ()
#6 0x000000000000010e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 15825):
#0 0x00007fed288d80c7 in ?? ()
#1 0x00006140000210a8 in ?? ()
#2 0x00007fece648bd90 in ?? ()
#3 0x00007fece648bdc0 in ?? ()
#4 0x00007fece648bec0 in ?? ()
#5 0x00007fece648bdb0 in ?? ()
#6 0x00007fece648be20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b082b in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007fece648bf40 in ?? ()
#11 0x00007fece648b67c in ?? ()
#12 0x00000032e648b5f0 in ?? ()
#13 0x00007fece5c8f000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 15824):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x00007fece6ca4f40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffc6612e070 in ?? ()
#5 0x00007fece6ca4f30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 15823):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x00007fece74bd100 in ?? ()
#2 0x000000000000008a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007fece74bd0f0 in ?? ()
#6 0x0000000000000114 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 15822):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 15821):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000007 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007fece84ece90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fece84eceb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007fed2fba3770 in ?? ()
#10 0x00007fece84eceb0 in ?? ()
#11 0x00000ffd9d09d9d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 15820):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000005 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007fece8d04e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fece8d04eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007fed2fba3770 in ?? ()
#10 0x00007fece8d04eb0 in ?? ()
#11 0x00000ffd9d1a09d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 15819):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 15818):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 15817):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 15816):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 15815):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 15814):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 15813):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 15811):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 15809):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 15807):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 15806):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 15805):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 15804):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 15803):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 15802):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 15801):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 15800):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 15799):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 15798):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 15797):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 15796):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 15795):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 15794):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 15793):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 15792):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 15791):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 15790):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 15789):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 15788):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 15787):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 15785):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 15784):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 15783):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 15782):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 15781):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 15779):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 15778):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 15776):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 15775):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007fecfc8bde90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fecfc8bdeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007fed2fba3770 in ?? ()
#10 0x00007fecfc8bdeb0 in ?? ()
#11 0x00000ffd9f917bd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 15774):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 15772):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 15771):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 15770):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 15769):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 15768):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 15767):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 15766):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 15765):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 15763):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 15761):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 15760):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 15759):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 15758):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 15757):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 15756):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 15755):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 15753):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 15752):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 15748):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007fed06a9ee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fed06a9eeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007fed2fba3770 in ?? ()
#10 0x00007fed06a9eeb0 in ?? ()
#11 0x00000ffda0d53dd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 15747):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 15746):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 15745):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 15744):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 15743):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 15742):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 15740):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 15739):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 15738):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 15737):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 15736):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 15735):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 15734):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 15733):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 15732):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 15730):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 15729):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 15728):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 15727):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 15726):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007fed10c7fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fed10c7feb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007fed2fba3770 in ?? ()
#10 0x00007fed10c7feb0 in ?? ()
#11 0x00000ffda218ffd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 15725):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 15724):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 15723):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 15722):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 15721):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 15720):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 15719):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 15717):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 15716):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 15715):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 15714):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 15713):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 15712):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 15711):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 15710):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 15708):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 15707):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 15706):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 15705):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 15703):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x00007fed1a669cc0 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019ca0 in ?? ()
#5 0x00007fed1a669cb0 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 15702):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 15700):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x00007fed1b69f240 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d1a8 in ?? ()
#5 0x00007fed1b69f230 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 15699):
#0 0x00007fed2fba3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 15698):
#0 0x00007fed288d6a47 in ?? ()
#1 0x00007fed1c6b2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007fed1c6b2310 in ?? ()
#4 0x00007fed1c6b2520 in ?? ()
#5 0x00007fed1c6b2360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007fed1c6b23e0 in ?? ()
#8 0x00007fed2af4925d in ?? ()
#9 0x3fb97455afe0c000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x00000000c4000000 in ?? ()
#14 0x00007fed00000000 in ?? ()
#15 0x41da16db2f4cc7ae in ?? ()
#16 0x00000ffe238ce47c in ?? ()
#17 0x00007fed1c6b23c0 in ?? ()
#18 0x00007fed2af4dba3 in ?? ()
#19 0x00007fed1c6b2390 in ?? ()
#20 0x3fb97455afe0c000 in ?? ()
#21 0x000000001c6b23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb97455afe0c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 15697):
#0 0x00007fed288d6a47 in ?? ()
#1 0x00007fed1ceb3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007fed1ceb3310 in ?? ()
#4 0x00007fed1ceb3520 in ?? ()
#5 0x00007fed1ceb3360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007fed1ceb33e0 in ?? ()
#8 0x00007fed2af4925d in ?? ()
#9 0x3fb9714f91cac000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000b2800000 in ?? ()
#14 0x00007fed00000000 in ?? ()
#15 0x41da16db2f4cc7ac in ?? ()
#16 0x00000ffe239ce67c in ?? ()
#17 0x00007fed1ceb33c0 in ?? ()
#18 0x00007fed2af4dba3 in ?? ()
#19 0x00007fed1ceb3390 in ?? ()
#20 0x3fb9714f91cac000 in ?? ()
#21 0x000000001ceb33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021098 in ?? ()
#24 0x3fb9714f91cac000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 15696):
#0 0x00007fed288d6a47 in ?? ()
#1 0x00007fed1d6b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007fed1d6b4310 in ?? ()
#4 0x00007fed1d6b4520 in ?? ()
#5 0x00007fed1d6b4360 in ?? ()
#6 0x0000614000020e98 in ?? ()
#7 0x00007fed1d6b43e0 in ?? ()
#8 0x00007fed2af4925d in ?? ()
#9 0x3fb95f8e5a6e8000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000c0c00000 in ?? ()
#14 0x00007fed00000000 in ?? ()
#15 0x41da16db2f4cc7bb in ?? ()
#16 0x00000ffe23ace87c in ?? ()
#17 0x00007fed1d6b43c0 in ?? ()
#18 0x00007fed2af4dba3 in ?? ()
#19 0x00007fed1d6b4390 in ?? ()
#20 0x3fb95f8e5a6e8000 in ?? ()
#21 0x000000001d6b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000020e98 in ?? ()
#24 0x3fb95f8e5a6e8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 15695):
#0 0x00007fed288d6a47 in ?? ()
#1 0x00007fed1f6ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007fed1f6ad310 in ?? ()
#4 0x00007fed1f6ad520 in ?? ()
#5 0x00007fed1f6ad360 in ?? ()
#6 0x0000614000020c98 in ?? ()
#7 0x00007fed1f6ad3e0 in ?? ()
#8 0x00007fed2af4925d in ?? ()
#9 0x3fb9768955994000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x000000004e400000 in ?? ()
#14 0x00007fed00000000 in ?? ()
#15 0x41da16db2f4cc7ab in ?? ()
#16 0x00000ffe23ecda7c in ?? ()
#17 0x00007fed1f6ad3c0 in ?? ()
#18 0x00007fed2af4dba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 15690):
#0 0x00007fed288c9cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fed210b77c8 in ?? ()
#3 0x000060200001fa10 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a8471 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 15688):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x00007fed208b60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007fed208b60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 15685):
#0 0x00007fed2fba79e2 in ?? ()
#1 0x00007fed200b3ba0 in ?? ()
#2 0x00007fed200b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fed200b3bf0 in ?? ()
#6 0x000000000177e760 in ?? ()
#7 0x00007fed200b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007fed200b3c40 in ?? ()
#10 0x00000000008a2967 in __sanitizer::theDepot ()
#11 0x00007fed354badcc in ?? ()
#12 0x00007fed354aa7c9 in ?? ()
#13 0x00007fed354aedb6 in ?? ()
#14 0x00007fed354b37f0 in ?? ()
#15 0x00007fed354b3619 in ?? ()
#16 0x0000000000a751bd in __sanitizer::theDepot ()
#17 0x00007fed2c77ef1e in ?? ()
#18 0x00007fed2fb9d6db in ?? ()
#19 0x00000ffe2400e684 in ?? ()
#20 0x00007fed200b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 15677):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x00007fed220b8c80 in ?? ()
#2 0x00000000000002bf in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007fed220b8c70 in ?? ()
#6 0x000000000000057e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 15673):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 15672):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 15671):
#0 0x00007fed2fba3fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007fed238bc450 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 15647):
#0 0x00007fed2fba7d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250625 03:47:19.099403 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 14196
I20250625 03:47:19.130728 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replaying log segment 1/1 (5.99M/6.60M this segment, stats: ops{read=3562 overwritten=0 applied=3562 ignored=0} inserts{seen=178000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250625 03:47:19.166301 14343 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 03:47:19.446063 16804 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:47:19.450474 16804 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:47:19.486030 16806 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:47:19.492451 16805 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:47:19.496461 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:47:19.501398 14840 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:47:19.502846 16806 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:47:19.510869 16805 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:47:19.508970 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:47:19.519084 14751 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d275425c3ef443f88abb12d24f4c2f2f in term 4.
I20250625 03:47:19.517000 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:47:19.527261 14132 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d275425c3ef443f88abb12d24f4c2f2f in term 4.
I20250625 03:47:19.533418 14841 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4, d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f
I20250625 03:47:19.539324 16804 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:47:19.544591 16807 raft_consensus.cc:2802] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250625 03:47:19.547950 16807 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9cb1b4dfaea74515a9e5aab1fbc0870f)
I20250625 03:47:19.548697 16807 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 4 FOLLOWER]: Advancing to term 5
I20250625 03:47:19.565433 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:47:19.568360 14682 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:47:19.572357 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4" is_pre_election: true
I20250625 03:47:19.573662 14132 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513d19d66cd54765940762233636fd31 in term 4.
I20250625 03:47:19.577991 14906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "513d19d66cd54765940762233636fd31" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:47:19.581465 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f" is_pre_election: true
W20250625 03:47:19.584219 14063 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:47:19.588773 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31" is_pre_election: true
I20250625 03:47:19.590137 14751 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975eeb611762460bb2f09f5b4b71e3d4 in term 4.
I20250625 03:47:19.592331 16806 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:47:19.596258 14905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "975eeb611762460bb2f09f5b4b71e3d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "d275425c3ef443f88abb12d24f4c2f2f" is_pre_election: true
I20250625 03:47:19.594750 16805 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
I20250625 03:47:19.603317 16807 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:47:19.614918 14906 raft_consensus.cc:2391] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 513d19d66cd54765940762233636fd31 in current term 5: Already voted for candidate d275425c3ef443f88abb12d24f4c2f2f in this term.
I20250625 03:47:19.622438 16807 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 election: Requested vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057)
I20250625 03:47:19.622308 15933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "506230f3ca974bf796583a6be750e38f"
I20250625 03:47:19.626331 14751 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "513d19d66cd54765940762233636fd31"
I20250625 03:47:19.628438 14132 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9126b7f95c9d434cb308581a7ce5b908" candidate_uuid: "d275425c3ef443f88abb12d24f4c2f2f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5012 } } ignore_live_leader: false dest_uuid: "975eeb611762460bb2f09f5b4b71e3d4"
I20250625 03:47:19.629700 14132 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 4 FOLLOWER]: Advancing to term 5
I20250625 03:47:19.627535 14751 raft_consensus.cc:3058] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 4 FOLLOWER]: Advancing to term 5
W20250625 03:47:19.633358 14840 leader_election.cc:343] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 election: Tablet error from VoteRequest() call to peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Illegal state: must be running to vote when last-logged opid is not known
I20250625 03:47:19.696480 14132 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d275425c3ef443f88abb12d24f4c2f2f in term 5.
I20250625 03:47:19.700083 14751 raft_consensus.cc:2466] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d275425c3ef443f88abb12d24f4c2f2f in term 5.
I20250625 03:47:19.701035 14839 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4, d275425c3ef443f88abb12d24f4c2f2f; no voters: 506230f3ca974bf796583a6be750e38f
I20250625 03:47:19.701779 16807 raft_consensus.cc:2802] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 5 FOLLOWER]: Leader election won for term 5
I20250625 03:47:19.702113 16807 raft_consensus.cc:695] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [term 5 LEADER]: Becoming Leader. State: Replica: d275425c3ef443f88abb12d24f4c2f2f, State: Running, Role: LEADER
I20250625 03:47:19.703773 16807 consensus_queue.cc:237] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5012, Committed index: 5012, Last appended: 4.5012, Last appended by leader: 5012, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:47:19.754048 13023 catalog_manager.cc:5582] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f reported cstate change: term changed from 4 to 5, leader changed from 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129) to d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132). New cstate: current_term: 5 leader_uuid: "d275425c3ef443f88abb12d24f4c2f2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } health_report { overall_health: HEALTHY } } }
I20250625 03:47:19.794174 14132 raft_consensus.cc:1273] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 5 FOLLOWER]: Refusing update from remote peer d275425c3ef443f88abb12d24f4c2f2f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5012. Preceding OpId from leader: term: 5 index: 5013. (index mismatch)
I20250625 03:47:19.796906 16807 consensus_queue.cc:1035] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5013, Last known committed idx: 5012, Time since last communication: 0.000s
W20250625 03:47:19.800973 14840 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250625 03:47:19.808312 14751 raft_consensus.cc:1273] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 5 FOLLOWER]: Refusing update from remote peer d275425c3ef443f88abb12d24f4c2f2f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5012. Preceding OpId from leader: term: 5 index: 5013. (index mismatch)
I20250625 03:47:19.812268 16807 consensus_queue.cc:1035] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5013, Last known committed idx: 5012, Time since last communication: 0.001s
W20250625 03:47:19.873072 14838 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.175.129:45233 timed out after 0.400s (SENT)
W20250625 03:47:19.898758 14061 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.175.129:45233 timed out after 0.348s (SENT)
W20250625 03:47:19.923905 14680 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.12.175.129:45233 timed out after 0.366s (SENT)
W20250625 03:47:20.106088 12990 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 14196: Timed out: CheckLeaks RPC to 127.12.175.129:45233 timed out after 1.000s (SENT)
I20250625 03:47:20.108188 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 14196
W20250625 03:47:20.270617 14061 leader_election.cc:336] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Timed out: RequestConsensusVote RPC to 127.12.175.129:45233 timed out after 0.348s (SENT)
I20250625 03:47:20.272576 14061 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:47:20.273554 16805 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 975eeb611762460bb2f09f5b4b71e3d4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250625 03:47:20.287235 14838 leader_election.cc:336] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Timed out: RequestConsensusVote RPC to 127.12.175.129:45233 timed out after 0.400s (SENT)
W20250625 03:47:20.307118 14680 leader_election.cc:336] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Timed out: RequestConsensusVote RPC to 127.12.175.129:45233 timed out after 0.366s (SENT)
I20250625 03:47:20.307907 14680 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4; no voters: 506230f3ca974bf796583a6be750e38f, 9cb1b4dfaea74515a9e5aab1fbc0870f, d275425c3ef443f88abb12d24f4c2f2f
I20250625 03:47:20.312266 16806 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 513d19d66cd54765940762233636fd31 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250625 03:47:20.330282 14840 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250625 03:47:20.413614 14838 leader_election.cc:336] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Timed out: RequestConsensusVote RPC to 127.12.175.129:45233 timed out after 0.396s (SENT)
W20250625 03:47:20.518679 14838 connection.cc:537] client connection to 127.12.175.129:45233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 03:47:20.526793 14061 connection.cc:537] client connection to 127.12.175.129:45233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 03:47:20.530025 14680 connection.cc:537] client connection to 127.12.175.129:45233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 03:47:20.535471 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 14038
W20250625 03:47:20.566334 14838 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 03:47:20.634474 14176 generic_service.cc:196] Checking for leaks (request via RPC)
W20250625 03:47:20.826745 14840 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250625 03:47:20.969199 14838 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250625 03:47:21.299515 14840 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
W20250625 03:47:21.519451 14838 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250625 03:47:21.539234 12990 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 14038: Timed out: CheckLeaks RPC to 127.12.175.130:34057 timed out after 1.000s (SENT)
I20250625 03:47:21.540831 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 14038
W20250625 03:47:21.786427 14841 connection.cc:537] client connection to 127.12.175.130:34057 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 03:47:21.790030 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 14657
W20250625 03:47:21.792356 14841 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): Couldn't send request to peer 975eeb611762460bb2f09f5b4b71e3d4. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250625 03:47:21.799839 14840 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
I20250625 03:47:21.855818 14797 generic_service.cc:196] Checking for leaks (request via RPC)
W20250625 03:47:22.039454 14838 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250625 03:47:22.287768 14841 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): Couldn't send request to peer 975eeb611762460bb2f09f5b4b71e3d4. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250625 03:47:22.355482 14840 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
W20250625 03:47:22.584395 14838 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250625 03:47:22.754438 14841 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): Couldn't send request to peer 975eeb611762460bb2f09f5b4b71e3d4. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250625 03:47:22.794312 12990 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 14657: Timed out: CheckLeaks RPC to 127.12.175.131:40037 timed out after 1.000s (SENT)
I20250625 03:47:22.795213 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 14657
W20250625 03:47:22.808714 14840 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 506230f3ca974bf796583a6be750e38f (127.12.175.133:36385): Couldn't send request to peer 506230f3ca974bf796583a6be750e38f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
W20250625 03:47:23.002569 14839 connection.cc:537] client connection to 127.12.175.131:40037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 03:47:23.006801 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 14819
W20250625 03:47:23.007664 14839 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Couldn't send request to peer 513d19d66cd54765940762233636fd31. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250625 03:47:23.057376 14838 consensus_peers.cc:487] T 9126b7f95c9d434cb308581a7ce5b908 P d275425c3ef443f88abb12d24f4c2f2f -> Peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Couldn't send request to peer 9cb1b4dfaea74515a9e5aab1fbc0870f. Status: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20250625 03:47:23.063006 14948 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 03:47:23.683437 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3920 overwritten=0 applied=3918 ignored=0} inserts{seen=195800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250625 03:47:23.686556 15869 tablet_bootstrap.cc:492] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Bootstrap complete.
I20250625 03:47:23.691097 15869 ts_tablet_manager.cc:1397] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Time spent bootstrapping tablet: real 139.581s user 49.961s sys 2.591s
I20250625 03:47:23.707808 15869 raft_consensus.cc:357] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 4 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:47:23.721567 15869 raft_consensus.cc:738] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506230f3ca974bf796583a6be750e38f, State: Initialized, Role: FOLLOWER
I20250625 03:47:23.723959 15869 consensus_queue.cc:260] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3918, Last appended: 4.3920, Last appended by leader: 3920, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
I20250625 03:47:23.728215 15869 ts_tablet_manager.cc:1428] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f: Time spent starting tablet: real 0.037s user 0.032s sys 0.008s
I20250625 03:47:23.950333 16834 raft_consensus.cc:491] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 03:47:23.951256 16834 raft_consensus.cc:513] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb1b4dfaea74515a9e5aab1fbc0870f" member_type: VOTER last_known_addr { host: "127.12.175.129" port: 45233 } } peers { permanent_uuid: "506230f3ca974bf796583a6be750e38f" member_type: VOTER last_known_addr { host: "127.12.175.133" port: 36385 } } peers { permanent_uuid: "513d19d66cd54765940762233636fd31" member_type: VOTER last_known_addr { host: "127.12.175.131" port: 40037 } } peers { permanent_uuid: "975eeb611762460bb2f09f5b4b71e3d4" member_type: VOTER last_known_addr { host: "127.12.175.130" port: 34057 } } peers { permanent_uuid: "d275425c3ef443f88abb12d24f4c2f2f" member_type: VOTER last_known_addr { host: "127.12.175.132" port: 42473 } }
W20250625 03:47:23.967185 15863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111)
I20250625 03:47:23.973594 16834 leader_election.cc:290] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233), 513d19d66cd54765940762233636fd31 (127.12.175.131:40037), 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057), d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473)
W20250625 03:47:23.979223 15866 leader_election.cc:336] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 975eeb611762460bb2f09f5b4b71e3d4 (127.12.175.130:34057): Network error: Client connection negotiation failed: client connection to 127.12.175.130:34057: connect: Connection refused (error 111)
W20250625 03:47:23.979936 15863 leader_election.cc:336] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9cb1b4dfaea74515a9e5aab1fbc0870f (127.12.175.129:45233): Network error: Client connection negotiation failed: client connection to 127.12.175.129:45233: connect: Connection refused (error 111)
W20250625 03:47:23.980320 15864 leader_election.cc:336] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 513d19d66cd54765940762233636fd31 (127.12.175.131:40037): Network error: Client connection negotiation failed: client connection to 127.12.175.131:40037: connect: Connection refused (error 111)
I20250625 03:47:23.980791 15864 leader_election.cc:304] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 506230f3ca974bf796583a6be750e38f; no voters: 513d19d66cd54765940762233636fd31, 975eeb611762460bb2f09f5b4b71e3d4, 9cb1b4dfaea74515a9e5aab1fbc0870f
I20250625 03:47:23.981977 16834 raft_consensus.cc:2747] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250625 03:47:24.010118 12990 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 14819: Timed out: CheckLeaks RPC to 127.12.175.132:42473 timed out after 1.000s (SENT)
I20250625 03:47:24.011235 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 14819
I20250625 03:47:24.138796 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 15840
W20250625 03:47:24.142277 16836 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 03:47:23.971145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.175.132:42473 (local address 127.12.175.133:57365)
0625 03:47:23.971894 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0625 03:47:23.972000 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0625 03:47:23.972939 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 03:47:24.139520 (+166581us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.175.132:42473: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":350}
W20250625 03:47:24.145583 15866 leader_election.cc:336] T 9126b7f95c9d434cb308581a7ce5b908 P 506230f3ca974bf796583a6be750e38f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d275425c3ef443f88abb12d24f4c2f2f (127.12.175.132:42473): Network error: Client connection negotiation failed: client connection to 127.12.175.132:42473: connect: Connection refused (error 111)
I20250625 03:47:24.162297 15977 generic_service.cc:196] Checking for leaks (request via RPC)
W20250625 03:47:25.142416 12990 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 15840: Timed out: CheckLeaks RPC to 127.12.175.133:36385 timed out after 1.000s (SENT)
I20250625 03:47:25.143209 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 15840
I20250625 03:47:25.268193 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 15647
I20250625 03:47:25.288164 15820 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 03:47:25.836656 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 15647
W20250625 03:47:25.887646 13014 connection.cc:537] server connection from 127.12.175.134:47829 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 03:47:25.888638 12990 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 12998
I20250625 03:47:25.908972 13059 generic_service.cc:196] Checking for leaks (request via RPC)
W20250625 03:47:26.892122 12990 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu pid 12998: Timed out: CheckLeaks RPC to 127.12.175.190:43735 timed out after 1.000s (SENT)
I20250625 03:47:26.893252 12990 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskU8fiuI/build/asan/bin/kudu with pid 12998
2025-06-25T03:47:27Z chronyd exiting
I20250625 03:47:27.024590 12990 test_util.cc:183] -----------------------------------------------
I20250625 03:47:27.024749 12990 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskU8fiuI/test-tmp/maintenance_mode-itest.5.RollingRestartArgs_RollingRestartITest.TestWorkloads_1.1750822977216960-12990-0
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/1, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 06-00 00-00 BE-BE BE-BE A0-0C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-0C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-0F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 02-00 00-00 05-00 00-00 00-BE BE-BE BE-BE BE-BE> (269728 ms)
[----------] 1 test from RollingRestartArgs/RollingRestartITest (269728 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (269729 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/1, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 06-00 00-00 BE-BE BE-BE A0-0C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-0C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-0F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 02-00 00-00 05-00 00-00 00-BE BE-BE BE-BE BE-BE>
1 FAILED TEST