[==========] Running 14 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 12 tests from RaftConsensusElectionITest
[ RUN ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240506 11:37:54.556972 10422 test_util.cc:268] Using random seed: -2146605052
I20240506 11:37:54.566713 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:37:54.566782 10422 ts_itest-base.cc:113] --------------
I20240506 11:37:54.566817 10422 ts_itest-base.cc:114] 3 tablet servers
I20240506 11:37:54.566846 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:37:54.566884 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:37:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:37:54Z Disabled control of system clock
I20240506 11:37:54.588500 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:41485
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:41485
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20240506 11:37:54.886126 10434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:37:54.886394 10434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:37:54.886466 10434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:37:54.892959 10434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:37:54.893047 10434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:37:54.893096 10434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:37:54.893148 10434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:37:54.901185 10434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:41485
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:41485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:37:54.902390 10434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:37:54.903986 10434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:37:54.914122 10439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:37:54.914914 10442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:37:54.915578 10440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:37:54.916347 10434 server_base.cc:1034] running on GCE node
I20240506 11:37:54.917086 10434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:37:54.918382 10434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:37:54.919585 10434 hybrid_clock.cc:648] HybridClock initialized: now 1714995474919503 us; error 51 us; skew 500 ppm
I20240506 11:37:54.919911 10434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:37:54.927412 10434 webserver.cc:460] Webserver started at http://127.10.45.190:39061/ using document root <none> and password file <none>
I20240506 11:37:54.928092 10434 fs_manager.cc:359] Metadata directory not provided
I20240506 11:37:54.928210 10434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:37:54.928507 10434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:37:54.930917 10434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/data/instance:
uuid: "2e81ba7cda444e028d1625fbe349146b"
format_stamp: "Formatted at 2024-05-06 11:37:54 on dist-test-slave-7l0g"
I20240506 11:37:54.931708 10434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/wal/instance:
uuid: "2e81ba7cda444e028d1625fbe349146b"
format_stamp: "Formatted at 2024-05-06 11:37:54 on dist-test-slave-7l0g"
I20240506 11:37:54.936203 10434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:37:54.939206 10448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:37:54.940440 10434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:37:54.940632 10434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/wal
uuid: "2e81ba7cda444e028d1625fbe349146b"
format_stamp: "Formatted at 2024-05-06 11:37:54 on dist-test-slave-7l0g"
I20240506 11:37:54.940811 10434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:37:54.952947 10434 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:37:54.953639 10434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:37:54.953884 10434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:37:54.970000 10434 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:41485
I20240506 11:37:54.970027 10499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:41485 every 8 connection(s)
I20240506 11:37:54.971237 10434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
I20240506 11:37:54.974443 10500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:37:54.981856 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 10434
I20240506 11:37:54.982089 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/master-0/wal/instance
I20240506 11:37:54.982517 10500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b: Bootstrap starting.
I20240506 11:37:54.985460 10500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b: Neither blocks nor log segments found. Creating new log.
I20240506 11:37:54.986641 10500 log.cc:824] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b: Log is configured to *not* fsync() on all Append() calls
I20240506 11:37:54.989415 10500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b: No bootstrap required, opened a new log
I20240506 11:37:54.995743 10500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e81ba7cda444e028d1625fbe349146b" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41485 } }
I20240506 11:37:54.996173 10500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:37:54.996270 10500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e81ba7cda444e028d1625fbe349146b, State: Initialized, Role: FOLLOWER
I20240506 11:37:54.996788 10500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e81ba7cda444e028d1625fbe349146b" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41485 } }
I20240506 11:37:54.996953 10500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:37:54.997061 10500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:37:54.997221 10500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:37:54.999855 10500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e81ba7cda444e028d1625fbe349146b" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41485 } }
I20240506 11:37:55.000345 10500 leader_election.cc:302] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e81ba7cda444e028d1625fbe349146b; no voters:
I20240506 11:37:55.001269 10500 leader_election.cc:288] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:37:55.001410 10505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:37:55.002609 10505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [term 1 LEADER]: Becoming Leader. State: Replica: 2e81ba7cda444e028d1625fbe349146b, State: Running, Role: LEADER
I20240506 11:37:55.003091 10505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e81ba7cda444e028d1625fbe349146b" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41485 } }
I20240506 11:37:55.003239 10500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:37:55.005995 10506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e81ba7cda444e028d1625fbe349146b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e81ba7cda444e028d1625fbe349146b" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41485 } } }
I20240506 11:37:55.006302 10506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [sys.catalog]: This master's current role is: LEADER
I20240506 11:37:55.006762 10507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e81ba7cda444e028d1625fbe349146b. Latest consensus state: current_term: 1 leader_uuid: "2e81ba7cda444e028d1625fbe349146b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e81ba7cda444e028d1625fbe349146b" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41485 } } }
I20240506 11:37:55.007177 10507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b [sys.catalog]: This master's current role is: LEADER
I20240506 11:37:55.020056 10510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:37:55.024853 10510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:37:55.032233 10510 catalog_manager.cc:1347] Generated new cluster ID: 7c6cf0d85cd640bca4bbc926152daa0c
I20240506 11:37:55.032334 10510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:37:55.055263 10510 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:37:55.056311 10510 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:37:55.063555 10510 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2e81ba7cda444e028d1625fbe349146b: Generated new TSK 0
I20240506 11:37:55.064270 10510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:37:55.069730 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41485
--builtin_ntp_servers=127.10.45.148:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20240506 11:37:55.378763 10525 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:37:55.379148 10525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:37:55.379248 10525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:37:55.379366 10525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:37:55.386313 10525 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20240506 11:37:55.386412 10525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:37:55.386544 10525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:37:55.394742 10525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41485
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:37:55.395943 10525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:37:55.397492 10525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:37:55.408464 10530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:37:55.409195 10531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:37:55.409930 10533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:37:55.410744 10525 server_base.cc:1034] running on GCE node
I20240506 11:37:55.411345 10525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:37:55.412189 10525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:37:55.413398 10525 hybrid_clock.cc:648] HybridClock initialized: now 1714995475413316 us; error 61 us; skew 500 ppm
I20240506 11:37:55.413695 10525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:37:55.415817 10525 webserver.cc:460] Webserver started at http://127.10.45.129:45941/ using document root <none> and password file <none>
I20240506 11:37:55.416481 10525 fs_manager.cc:359] Metadata directory not provided
I20240506 11:37:55.416612 10525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:37:55.416885 10525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:37:55.419389 10525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/data/instance:
uuid: "f1336c0248474febb226d72011301bda"
format_stamp: "Formatted at 2024-05-06 11:37:55 on dist-test-slave-7l0g"
I20240506 11:37:55.420109 10525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/wal/instance:
uuid: "f1336c0248474febb226d72011301bda"
format_stamp: "Formatted at 2024-05-06 11:37:55 on dist-test-slave-7l0g"
I20240506 11:37:55.424852 10525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240506 11:37:55.427807 10539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:37:55.428988 10525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240506 11:37:55.429131 10525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/wal
uuid: "f1336c0248474febb226d72011301bda"
format_stamp: "Formatted at 2024-05-06 11:37:55 on dist-test-slave-7l0g"
I20240506 11:37:55.429278 10525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:37:55.445510 10525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:37:55.446251 10525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:37:55.446502 10525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:37:55.447356 10525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:37:55.448729 10525 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:37:55.448809 10525 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:37:55.448930 10525 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:37:55.448982 10525 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:37:55.479856 10525 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:38907
I20240506 11:37:55.479892 10651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:38907 every 8 connection(s)
I20240506 11:37:55.481063 10525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
I20240506 11:37:55.485373 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 10525
I20240506 11:37:55.485549 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-0/wal/instance
I20240506 11:37:55.488775 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41485
--builtin_ntp_servers=127.10.45.148:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20240506 11:37:55.493536 10652 heartbeater.cc:347] Connected to a master server at 127.10.45.190:41485
I20240506 11:37:55.493845 10652 heartbeater.cc:464] Registering TS with master...
I20240506 11:37:55.494614 10652 heartbeater.cc:511] Master 127.10.45.190:41485 requested a full tablet report, sending...
I20240506 11:37:55.496778 10465 ts_manager.cc:194] Registered new tserver with Master: f1336c0248474febb226d72011301bda (127.10.45.129:38907)
I20240506 11:37:55.498941 10465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:38913
W20240506 11:37:55.816455 10655 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:37:55.816797 10655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:37:55.816864 10655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:37:55.816939 10655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:37:55.824875 10655 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20240506 11:37:55.824985 10655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:37:55.825112 10655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:37:55.833966 10655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41485
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:37:55.835361 10655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:37:55.836988 10655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:37:55.848722 10661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:37:55.849442 10663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:37:55.850359 10660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:37:55.851202 10655 server_base.cc:1034] running on GCE node
I20240506 11:37:55.851799 10655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:37:55.852769 10655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:37:55.853986 10655 hybrid_clock.cc:648] HybridClock initialized: now 1714995475853921 us; error 39 us; skew 500 ppm
I20240506 11:37:55.854295 10655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:37:55.856809 10655 webserver.cc:460] Webserver started at http://127.10.45.130:43233/ using document root <none> and password file <none>
I20240506 11:37:55.857546 10655 fs_manager.cc:359] Metadata directory not provided
I20240506 11:37:55.857663 10655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:37:55.858718 10655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:37:55.861518 10655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/data/instance:
uuid: "42beb939ba7d4b7ab48832dfab44b689"
format_stamp: "Formatted at 2024-05-06 11:37:55 on dist-test-slave-7l0g"
I20240506 11:37:55.862331 10655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/wal/instance:
uuid: "42beb939ba7d4b7ab48832dfab44b689"
format_stamp: "Formatted at 2024-05-06 11:37:55 on dist-test-slave-7l0g"
I20240506 11:37:55.867801 10655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240506 11:37:55.871296 10669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:37:55.872689 10655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240506 11:37:55.872850 10655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/wal
uuid: "42beb939ba7d4b7ab48832dfab44b689"
format_stamp: "Formatted at 2024-05-06 11:37:55 on dist-test-slave-7l0g"
I20240506 11:37:55.873029 10655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:37:55.895684 10655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:37:55.896507 10655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:37:55.896792 10655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:37:55.897773 10655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:37:55.899497 10655 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:37:55.899580 10655 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:37:55.899722 10655 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:37:55.899777 10655 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:37:55.933739 10655 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:41291
I20240506 11:37:55.933813 10781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:41291 every 8 connection(s)
I20240506 11:37:55.935091 10655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
I20240506 11:37:55.936446 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 10655
I20240506 11:37:55.936607 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-1/wal/instance
I20240506 11:37:55.939922 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41485
--builtin_ntp_servers=127.10.45.148:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20240506 11:37:55.950351 10782 heartbeater.cc:347] Connected to a master server at 127.10.45.190:41485
I20240506 11:37:55.950665 10782 heartbeater.cc:464] Registering TS with master...
I20240506 11:37:55.951531 10782 heartbeater.cc:511] Master 127.10.45.190:41485 requested a full tablet report, sending...
I20240506 11:37:55.952917 10465 ts_manager.cc:194] Registered new tserver with Master: 42beb939ba7d4b7ab48832dfab44b689 (127.10.45.130:41291)
I20240506 11:37:55.953645 10465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:51033
W20240506 11:37:56.262388 10785 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:37:56.262756 10785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:37:56.262861 10785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:37:56.262997 10785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:37:56.270028 10785 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20240506 11:37:56.270152 10785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:37:56.270325 10785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:37:56.278729 10785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41485
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:37:56.279954 10785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:37:56.281555 10785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:37:56.292824 10790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:37:56.293668 10793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:37:56.294508 10791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:37:56.295302 10785 server_base.cc:1034] running on GCE node
I20240506 11:37:56.295826 10785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:37:56.296643 10785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:37:56.297840 10785 hybrid_clock.cc:648] HybridClock initialized: now 1714995476297782 us; error 36 us; skew 500 ppm
I20240506 11:37:56.298102 10785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:37:56.300267 10785 webserver.cc:460] Webserver started at http://127.10.45.131:40747/ using document root <none> and password file <none>
I20240506 11:37:56.300952 10785 fs_manager.cc:359] Metadata directory not provided
I20240506 11:37:56.301067 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:37:56.301342 10785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:37:56.303893 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/data/instance:
uuid: "53b53b7c13bc4beb96c16df7180790d4"
format_stamp: "Formatted at 2024-05-06 11:37:56 on dist-test-slave-7l0g"
I20240506 11:37:56.304637 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/wal/instance:
uuid: "53b53b7c13bc4beb96c16df7180790d4"
format_stamp: "Formatted at 2024-05-06 11:37:56 on dist-test-slave-7l0g"
I20240506 11:37:56.309306 10785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240506 11:37:56.312350 10799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:37:56.313514 10785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240506 11:37:56.313681 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/wal
uuid: "53b53b7c13bc4beb96c16df7180790d4"
format_stamp: "Formatted at 2024-05-06 11:37:56 on dist-test-slave-7l0g"
I20240506 11:37:56.313838 10785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:37:56.330081 10785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:37:56.330786 10785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:37:56.331079 10785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:37:56.331918 10785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:37:56.333400 10785 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:37:56.333472 10785 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:37:56.333586 10785 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:37:56.333644 10785 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:37:56.365423 10785 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:42305
I20240506 11:37:56.365460 10911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:42305 every 8 connection(s)
I20240506 11:37:56.366595 10785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
I20240506 11:37:56.369699 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 10785
I20240506 11:37:56.369882 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-2/wal/instance
I20240506 11:37:56.378489 10912 heartbeater.cc:347] Connected to a master server at 127.10.45.190:41485
I20240506 11:37:56.378746 10912 heartbeater.cc:464] Registering TS with master...
I20240506 11:37:56.379493 10912 heartbeater.cc:511] Master 127.10.45.190:41485 requested a full tablet report, sending...
I20240506 11:37:56.380672 10465 ts_manager.cc:194] Registered new tserver with Master: 53b53b7c13bc4beb96c16df7180790d4 (127.10.45.131:42305)
I20240506 11:37:56.381381 10465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:51023
I20240506 11:37:56.385496 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:37:56.388976 10422 test_util.cc:268] Using random seed: -2144773039
I20240506 11:37:56.410753 10465 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49760:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 11:37:56.412283 10465 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 11:37:56.445326 10717 tablet_service.cc:1467] Processing CreateTablet for tablet 4acb756732854fab8e039c874dbc69d3 (DEFAULT_TABLE table=test-workload [id=da403fff06ae4adeb5e25a7f356e0f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:37:56.446138 10847 tablet_service.cc:1467] Processing CreateTablet for tablet 4acb756732854fab8e039c874dbc69d3 (DEFAULT_TABLE table=test-workload [id=da403fff06ae4adeb5e25a7f356e0f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:37:56.446712 10717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acb756732854fab8e039c874dbc69d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:37:56.447567 10847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acb756732854fab8e039c874dbc69d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:37:56.448403 10587 tablet_service.cc:1467] Processing CreateTablet for tablet 4acb756732854fab8e039c874dbc69d3 (DEFAULT_TABLE table=test-workload [id=da403fff06ae4adeb5e25a7f356e0f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:37:56.449663 10587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4acb756732854fab8e039c874dbc69d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:37:56.456431 10930 tablet_bootstrap.cc:492] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689: Bootstrap starting.
I20240506 11:37:56.459334 10930 tablet_bootstrap.cc:654] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689: Neither blocks nor log segments found. Creating new log.
I20240506 11:37:56.460031 10931 tablet_bootstrap.cc:492] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4: Bootstrap starting.
I20240506 11:37:56.460592 10930 log.cc:824] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689: Log is configured to *not* fsync() on all Append() calls
I20240506 11:37:56.461675 10932 tablet_bootstrap.cc:492] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda: Bootstrap starting.
I20240506 11:37:56.464001 10931 tablet_bootstrap.cc:654] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4: Neither blocks nor log segments found. Creating new log.
I20240506 11:37:56.464737 10932 tablet_bootstrap.cc:654] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda: Neither blocks nor log segments found. Creating new log.
I20240506 11:37:56.465392 10931 log.cc:824] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4: Log is configured to *not* fsync() on all Append() calls
I20240506 11:37:56.465533 10932 log.cc:824] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda: Log is configured to *not* fsync() on all Append() calls
I20240506 11:37:56.468072 10930 tablet_bootstrap.cc:492] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689: No bootstrap required, opened a new log
I20240506 11:37:56.468134 10932 tablet_bootstrap.cc:492] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda: No bootstrap required, opened a new log
I20240506 11:37:56.468319 10930 ts_tablet_manager.cc:1367] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240506 11:37:56.468411 10932 ts_tablet_manager.cc:1367] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240506 11:37:56.468480 10931 tablet_bootstrap.cc:492] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4: No bootstrap required, opened a new log
I20240506 11:37:56.468705 10931 ts_tablet_manager.cc:1367] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240506 11:37:56.473838 10930 raft_consensus.cc:348] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } }
I20240506 11:37:56.474344 10930 raft_consensus.cc:729] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42beb939ba7d4b7ab48832dfab44b689, State: Initialized, Role: FOLLOWER
I20240506 11:37:56.474756 10930 consensus_queue.cc:260] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } }
I20240506 11:37:56.475741 10782 heartbeater.cc:503] Master 127.10.45.190:41485 was elected leader, sending a full tablet report...
I20240506 11:37:56.476135 10930 ts_tablet_manager.cc:1398] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240506 11:37:56.476536 10931 raft_consensus.cc:348] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } }
I20240506 11:37:56.476882 10931 raft_consensus.cc:729] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b53b7c13bc4beb96c16df7180790d4, State: Initialized, Role: FOLLOWER
I20240506 11:37:56.477118 10932 raft_consensus.cc:348] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } }
I20240506 11:37:56.477465 10932 raft_consensus.cc:729] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1336c0248474febb226d72011301bda, State: Initialized, Role: FOLLOWER
I20240506 11:37:56.477452 10931 consensus_queue.cc:260] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } }
I20240506 11:37:56.477860 10932 consensus_queue.cc:260] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } }
I20240506 11:37:56.478451 10912 heartbeater.cc:503] Master 127.10.45.190:41485 was elected leader, sending a full tablet report...
I20240506 11:37:56.485517 10652 heartbeater.cc:503] Master 127.10.45.190:41485 was elected leader, sending a full tablet report...
I20240506 11:37:56.478914 10931 ts_tablet_manager.cc:1398] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4: Time spent starting tablet: real 0.010s user 0.000s sys 0.011s
I20240506 11:37:56.486361 10932 ts_tablet_manager.cc:1398] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
W20240506 11:37:56.515285 10777 generic_service.cc:139] {username='slave'} at 127.0.0.1:58330 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20240506 11:37:56.515484 10777 generic_service.cc:175] {username='slave'} at 127.0.0.1:58330 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.10.45.130:41291'
W20240506 11:37:56.516733 10777 generic_service.cc:139] {username='slave'} at 127.0.0.1:58330 forcing change of non-runtime-safe flag fail_dns_resolution
I20240506 11:37:56.516824 10777 generic_service.cc:175] {username='slave'} at 127.0.0.1:58330 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20240506 11:37:56.521917 10907 generic_service.cc:139] {username='slave'} at 127.0.0.1:52560 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20240506 11:37:56.522063 10907 generic_service.cc:175] {username='slave'} at 127.0.0.1:52560 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.10.45.130:41291'
W20240506 11:37:56.523075 10907 generic_service.cc:139] {username='slave'} at 127.0.0.1:52560 forcing change of non-runtime-safe flag fail_dns_resolution
I20240506 11:37:56.523162 10907 generic_service.cc:175] {username='slave'} at 127.0.0.1:52560 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20240506 11:37:56.527776 10647 generic_service.cc:139] {username='slave'} at 127.0.0.1:58570 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20240506 11:37:56.527918 10647 generic_service.cc:175] {username='slave'} at 127.0.0.1:58570 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.10.45.130:41291'
W20240506 11:37:56.528960 10647 generic_service.cc:139] {username='slave'} at 127.0.0.1:58570 forcing change of non-runtime-safe flag fail_dns_resolution
I20240506 11:37:56.529040 10647 generic_service.cc:175] {username='slave'} at 127.0.0.1:58570 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20240506 11:37:56.530126 10867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4acb756732854fab8e039c874dbc69d3"
dest_uuid: "53b53b7c13bc4beb96c16df7180790d4"
from {username='slave'} at 127.0.0.1:52558
I20240506 11:37:56.530382 10867 raft_consensus.cc:482] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:37:56.530553 10867 raft_consensus.cc:3037] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:37:56.532579 10867 raft_consensus.cc:504] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } }
I20240506 11:37:56.534448 10867 leader_election.cc:288] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [CANDIDATE]: Term 1 election: Requested vote from peers 42beb939ba7d4b7ab48832dfab44b689 (127.10.45.130:41291), f1336c0248474febb226d72011301bda (127.10.45.129:38907)
W20240506 11:37:56.534896 10941 leader_election.cc:334] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 42beb939ba7d4b7ab48832dfab44b689 (127.10.45.130:41291): Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.45.130:41291
I20240506 11:37:56.540164 10607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4acb756732854fab8e039c874dbc69d3" candidate_uuid: "53b53b7c13bc4beb96c16df7180790d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f1336c0248474febb226d72011301bda"
I20240506 11:37:56.540488 10607 raft_consensus.cc:3037] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:37:56.542486 10607 raft_consensus.cc:2454] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b53b7c13bc4beb96c16df7180790d4 in term 1.
I20240506 11:37:56.543010 10801 leader_election.cc:302] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53b53b7c13bc4beb96c16df7180790d4, f1336c0248474febb226d72011301bda; no voters: 42beb939ba7d4b7ab48832dfab44b689
I20240506 11:37:56.543355 10937 raft_consensus.cc:2781] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:37:56.543815 10937 raft_consensus.cc:686] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [term 1 LEADER]: Becoming Leader. State: Replica: 53b53b7c13bc4beb96c16df7180790d4, State: Running, Role: LEADER
I20240506 11:37:56.544328 10937 consensus_queue.cc:237] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } }
I20240506 11:37:56.549649 10465 catalog_manager.cc:5484] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53b53b7c13bc4beb96c16df7180790d4 (127.10.45.131). New cstate: current_term: 1 leader_uuid: "53b53b7c13bc4beb96c16df7180790d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } health_report { overall_health: UNKNOWN } } }
W20240506 11:37:56.618093 10913 tablet.cc:2369] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 11:37:56.669925 10941 consensus_peers.cc:485] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 -> Peer 42beb939ba7d4b7ab48832dfab44b689 (127.10.45.130:41291): Couldn't send request to peer 42beb939ba7d4b7ab48832dfab44b689. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.45.130:41291. This is attempt 1: this message will repeat every 5th retry.
I20240506 11:37:56.672271 10937 consensus_queue.cc:1035] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 11:37:56.687078 10783 tablet.cc:2369] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 11:37:57.127862 10941 consensus_peers.cc:485] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 -> Peer 42beb939ba7d4b7ab48832dfab44b689 (127.10.45.130:41291): Couldn't send request to peer 42beb939ba7d4b7ab48832dfab44b689. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.45.130:41291. This is attempt 6: this message will repeat every 5th retry.
W20240506 11:37:57.674538 10941 consensus_peers.cc:485] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 -> Peer 42beb939ba7d4b7ab48832dfab44b689 (127.10.45.130:41291): Couldn't send request to peer 42beb939ba7d4b7ab48832dfab44b689. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.45.130:41291. This is attempt 11: this message will repeat every 5th retry.
W20240506 11:37:58.187523 10941 consensus_peers.cc:485] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 -> Peer 42beb939ba7d4b7ab48832dfab44b689 (127.10.45.130:41291): Couldn't send request to peer 42beb939ba7d4b7ab48832dfab44b689. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.45.130:41291. This is attempt 16: this message will repeat every 5th retry.
W20240506 11:37:58.681303 10941 consensus_peers.cc:485] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 -> Peer 42beb939ba7d4b7ab48832dfab44b689 (127.10.45.130:41291): Couldn't send request to peer 42beb939ba7d4b7ab48832dfab44b689. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.45.130:41291. This is attempt 21: this message will repeat every 5th retry.
W20240506 11:37:59.174144 10941 consensus_peers.cc:485] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 -> Peer 42beb939ba7d4b7ab48832dfab44b689 (127.10.45.130:41291): Couldn't send request to peer 42beb939ba7d4b7ab48832dfab44b689. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.45.130:41291. This is attempt 26: this message will repeat every 5th retry.
I20240506 11:37:59.569674 10937 consensus_queue.cc:579] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [LEADER]: Leader has been unable to successfully communicate with peer 42beb939ba7d4b7ab48832dfab44b689 for more than 3 seconds (3.025s)
W20240506 11:37:59.572695 10464 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4acb756732854fab8e039c874dbc69d3 with cas_config_opid_index -1: no extra replica candidate found for tablet 4acb756732854fab8e039c874dbc69d3 (table test-workload [id=da403fff06ae4adeb5e25a7f356e0f06]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20240506 11:37:59.622423 10777 generic_service.cc:139] {username='slave'} at 127.0.0.1:58330 forcing change of non-runtime-safe flag fail_dns_resolution
I20240506 11:37:59.622566 10777 generic_service.cc:175] {username='slave'} at 127.0.0.1:58330 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20240506 11:37:59.623556 10907 generic_service.cc:139] {username='slave'} at 127.0.0.1:52560 forcing change of non-runtime-safe flag fail_dns_resolution
I20240506 11:37:59.623659 10907 generic_service.cc:175] {username='slave'} at 127.0.0.1:52560 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20240506 11:37:59.624619 10647 generic_service.cc:139] {username='slave'} at 127.0.0.1:58570 forcing change of non-runtime-safe flag fail_dns_resolution
I20240506 11:37:59.624727 10647 generic_service.cc:175] {username='slave'} at 127.0.0.1:58570 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
I20240506 11:37:59.655007 10737 raft_consensus.cc:3037] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:37:59.784998 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.132:0
--local_ip_for_outbound_sockets=127.10.45.132
--webserver_interface=127.10.45.132
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41485
--builtin_ntp_servers=127.10.45.148:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20240506 11:38:00.090389 10954 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:38:00.090823 10954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:38:00.090939 10954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:38:00.091059 10954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:38:00.097959 10954 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20240506 11:38:00.098083 10954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:38:00.098276 10954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.132
I20240506 11:38:00.106997 10954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.45.132
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41485
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.10.45.132
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:38:00.108286 10954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:38:00.109918 10954 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:38:00.121431 10959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:00.122226 10960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:00.123046 10962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:38:00.123857 10954 server_base.cc:1034] running on GCE node
I20240506 11:38:00.124450 10954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:38:00.125344 10954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:38:00.126567 10954 hybrid_clock.cc:648] HybridClock initialized: now 1714995480126506 us; error 36 us; skew 500 ppm
I20240506 11:38:00.126880 10954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:38:00.134848 10954 webserver.cc:460] Webserver started at http://127.10.45.132:33689/ using document root <none> and password file <none>
I20240506 11:38:00.135632 10954 fs_manager.cc:359] Metadata directory not provided
I20240506 11:38:00.135758 10954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:38:00.136065 10954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:38:00.138582 10954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/data/instance:
uuid: "c56c4dc25438482c8576acc9f7a3386e"
format_stamp: "Formatted at 2024-05-06 11:38:00 on dist-test-slave-7l0g"
I20240506 11:38:00.139318 10954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/wal/instance:
uuid: "c56c4dc25438482c8576acc9f7a3386e"
format_stamp: "Formatted at 2024-05-06 11:38:00 on dist-test-slave-7l0g"
I20240506 11:38:00.144372 10954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240506 11:38:00.147454 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:00.148733 10954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240506 11:38:00.148882 10954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/wal
uuid: "c56c4dc25438482c8576acc9f7a3386e"
format_stamp: "Formatted at 2024-05-06 11:38:00 on dist-test-slave-7l0g"
I20240506 11:38:00.149044 10954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:38:00.171384 10954 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:38:00.172138 10954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:38:00.172415 10954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:38:00.173281 10954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:38:00.174661 10954 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:38:00.174742 10954 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:00.174866 10954 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:38:00.174919 10954 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:00.205035 10954 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.132:42803
I20240506 11:38:00.205098 11080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.132:42803 every 8 connection(s)
I20240506 11:38:00.206593 10954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
I20240506 11:38:00.210278 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 10954
I20240506 11:38:00.210449 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0/raft_consensus_election-itest-cluster/ts-3/wal/instance
I20240506 11:38:00.218988 11081 heartbeater.cc:347] Connected to a master server at 127.10.45.190:41485
I20240506 11:38:00.219236 11081 heartbeater.cc:464] Registering TS with master...
I20240506 11:38:00.219971 11081 heartbeater.cc:511] Master 127.10.45.190:41485 requested a full tablet report, sending...
I20240506 11:38:00.221212 10464 ts_manager.cc:194] Registered new tserver with Master: c56c4dc25438482c8576acc9f7a3386e (127.10.45.132:42803)
I20240506 11:38:00.221954 10464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.132:38395
I20240506 11:38:00.239918 10867 consensus_queue.cc:237] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } } peers { permanent_uuid: "c56c4dc25438482c8576acc9f7a3386e" member_type: NON_VOTER last_known_addr { host: "127.10.45.132" port: 42803 } attrs { promote: true } }
I20240506 11:38:00.242568 10607 raft_consensus.cc:1261] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda [term 1 FOLLOWER]: Refusing update from remote peer 53b53b7c13bc4beb96c16df7180790d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:38:00.242892 10737 raft_consensus.cc:1261] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689 [term 1 FOLLOWER]: Refusing update from remote peer 53b53b7c13bc4beb96c16df7180790d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:38:00.243240 10937 consensus_queue.cc:1035] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 11:38:00.243557 11086 consensus_queue.cc:1035] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 11:38:00.246625 10937 raft_consensus.cc:2932] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c56c4dc25438482c8576acc9f7a3386e (127.10.45.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } } peers { permanent_uuid: "c56c4dc25438482c8576acc9f7a3386e" member_type: NON_VOTER last_known_addr { host: "127.10.45.132" port: 42803 } attrs { promote: true } } }
I20240506 11:38:00.247277 10737 raft_consensus.cc:2932] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c56c4dc25438482c8576acc9f7a3386e (127.10.45.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } } peers { permanent_uuid: "c56c4dc25438482c8576acc9f7a3386e" member_type: NON_VOTER last_known_addr { host: "127.10.45.132" port: 42803 } attrs { promote: true } } }
I20240506 11:38:00.247574 10607 raft_consensus.cc:2932] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c56c4dc25438482c8576acc9f7a3386e (127.10.45.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } } peers { permanent_uuid: "c56c4dc25438482c8576acc9f7a3386e" member_type: NON_VOTER last_known_addr { host: "127.10.45.132" port: 42803 } attrs { promote: true } } }
I20240506 11:38:00.251253 10452 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4acb756732854fab8e039c874dbc69d3 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20240506 11:38:00.254093 10464 catalog_manager.cc:5484] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 reported cstate change: config changed from index -1 to 2, NON_VOTER c56c4dc25438482c8576acc9f7a3386e (127.10.45.132) added. New cstate: current_term: 1 leader_uuid: "53b53b7c13bc4beb96c16df7180790d4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c56c4dc25438482c8576acc9f7a3386e" member_type: NON_VOTER last_known_addr { host: "127.10.45.132" port: 42803 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240506 11:38:00.255739 10802 consensus_peers.cc:485] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 -> Peer c56c4dc25438482c8576acc9f7a3386e (127.10.45.132:42803): Couldn't send request to peer c56c4dc25438482c8576acc9f7a3386e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4acb756732854fab8e039c874dbc69d3. This is attempt 1: this message will repeat every 5th retry.
I20240506 11:38:00.259621 10867 consensus_queue.cc:237] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } }
I20240506 11:38:00.261237 10737 raft_consensus.cc:1261] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689 [term 1 FOLLOWER]: Refusing update from remote peer 53b53b7c13bc4beb96c16df7180790d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 11:38:00.261359 10607 raft_consensus.cc:1261] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda [term 1 FOLLOWER]: Refusing update from remote peer 53b53b7c13bc4beb96c16df7180790d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 11:38:00.261801 11090 consensus_queue.cc:1035] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:38:00.262004 10937 consensus_queue.cc:1035] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:38:00.264300 11086 raft_consensus.cc:2932] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER c56c4dc25438482c8576acc9f7a3386e (127.10.45.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } } }
I20240506 11:38:00.264851 10607 raft_consensus.cc:2932] T 4acb756732854fab8e039c874dbc69d3 P f1336c0248474febb226d72011301bda [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER c56c4dc25438482c8576acc9f7a3386e (127.10.45.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } } }
I20240506 11:38:00.264953 10737 raft_consensus.cc:2932] T 4acb756732854fab8e039c874dbc69d3 P 42beb939ba7d4b7ab48832dfab44b689 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER c56c4dc25438482c8576acc9f7a3386e (127.10.45.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } } }
I20240506 11:38:00.267010 10452 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4acb756732854fab8e039c874dbc69d3 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240506 11:38:00.268028 10777 generic_service.cc:139] {username='slave'} at 127.0.0.1:58330 forcing change of non-runtime-safe flag fail_dns_resolution
I20240506 11:38:00.268169 10777 generic_service.cc:175] {username='slave'} at 127.0.0.1:58330 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20240506 11:38:00.268045 10465 catalog_manager.cc:5484] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 reported cstate change: config changed from index 2 to 3, NON_VOTER c56c4dc25438482c8576acc9f7a3386e (127.10.45.132) evicted. New cstate: current_term: 1 leader_uuid: "53b53b7c13bc4beb96c16df7180790d4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42beb939ba7d4b7ab48832dfab44b689" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53b53b7c13bc4beb96c16df7180790d4" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1336c0248474febb226d72011301bda" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 38907 } health_report { overall_health: HEALTHY } } }
W20240506 11:38:00.269645 10907 generic_service.cc:139] {username='slave'} at 127.0.0.1:52560 forcing change of non-runtime-safe flag fail_dns_resolution
I20240506 11:38:00.269752 10907 generic_service.cc:175] {username='slave'} at 127.0.0.1:52560 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20240506 11:38:00.270869 10647 generic_service.cc:139] {username='slave'} at 127.0.0.1:58570 forcing change of non-runtime-safe flag fail_dns_resolution
I20240506 11:38:00.271023 10647 generic_service.cc:175] {username='slave'} at 127.0.0.1:58570 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20240506 11:38:00.272260 10867 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4acb756732854fab8e039c874dbc69d3"
dest_uuid: "53b53b7c13bc4beb96c16df7180790d4"
mode: GRACEFUL
new_leader_uuid: "53b53b7c13bc4beb96c16df7180790d4"
from {username='slave'} at 127.0.0.1:52558
I20240506 11:38:00.272516 10867 raft_consensus.cc:595] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 [term 1 LEADER]: Received request to transfer leadership to TS 53b53b7c13bc4beb96c16df7180790d4
I20240506 11:38:00.279265 11016 tablet_service.cc:1514] Processing DeleteTablet for tablet 4acb756732854fab8e039c874dbc69d3 with delete_type TABLET_DATA_TOMBSTONED (TS c56c4dc25438482c8576acc9f7a3386e not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45124
W20240506 11:38:00.285471 10451 catalog_manager.cc:4797] TS c56c4dc25438482c8576acc9f7a3386e (127.10.45.132:42803): delete failed for tablet 4acb756732854fab8e039c874dbc69d3 because the tablet was not found. No further retry: Not found: Tablet not found: 4acb756732854fab8e039c874dbc69d3
I20240506 11:38:01.224191 11081 heartbeater.cc:503] Master 127.10.45.190:41485 was elected leader, sending a full tablet report...
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/raft_consensus_election-itest.cc:291: Failure
Value of: tablets.empty()
Actual: true
Expected: false
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
I20240506 11:38:30.361675 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 10525
I20240506 11:38:30.362915 10647 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:38:30.445219 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 10525
I20240506 11:38:30.464491 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 10655
I20240506 11:38:30.465463 10777 generic_service.cc:196] Checking for leaks (request via RPC)
W20240506 11:38:30.498836 10801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.45.129:38907: connect: Connection refused (error 111)
W20240506 11:38:30.500172 10801 consensus_peers.cc:485] T 4acb756732854fab8e039c874dbc69d3 P 53b53b7c13bc4beb96c16df7180790d4 -> Peer f1336c0248474febb226d72011301bda (127.10.45.129:38907): Couldn't send request to peer f1336c0248474febb226d72011301bda. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:38907: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:38:30.549005 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 10655
I20240506 11:38:30.571864 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 10785
I20240506 11:38:30.572844 10907 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:38:30.658542 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 10785
I20240506 11:38:30.673245 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 10954
I20240506 11:38:30.679021 11076 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:38:30.756197 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 10954
I20240506 11:38:30.772552 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 10434
I20240506 11:38:30.773594 10495 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:38:30.872330 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 10434
2024-05-06T11:38:30Z chronyd exiting
I20240506 11:38:30.905649 10422 test_util.cc:175] -----------------------------------------------
I20240506 11:38:30.905746 10422 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1714995474491875-10422-0
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers (36352 ms)
[ RUN ] RaftConsensusElectionITest.RunLeaderElection
I20240506 11:38:30.907613 10422 test_util.cc:268] Using random seed: -2110254408
I20240506 11:38:30.909256 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:38:30.909325 10422 ts_itest-base.cc:113] --------------
I20240506 11:38:30.909374 10422 ts_itest-base.cc:114] 3 tablet servers
I20240506 11:38:30.909441 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:38:30.909489 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:38:30Z Disabled control of system clock
I20240506 11:38:30.932130 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:45417
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:45417 with env {}
W20240506 11:38:31.244166 11123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:38:31.244447 11123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:38:31.244602 11123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:38:31.251545 11123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:38:31.251652 11123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:38:31.251708 11123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:38:31.251791 11123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:38:31.260164 11123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:45417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:45417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:38:31.261377 11123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:38:31.263001 11123 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:38:31.274219 11128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:31.275110 11129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:31.275923 11131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:38:31.276819 11123 server_base.cc:1034] running on GCE node
I20240506 11:38:31.277590 11123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:38:31.278929 11123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:38:31.280189 11123 hybrid_clock.cc:648] HybridClock initialized: now 1714995511280119 us; error 46 us; skew 500 ppm
I20240506 11:38:31.280494 11123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:38:31.282631 11123 webserver.cc:460] Webserver started at http://127.10.45.190:39833/ using document root <none> and password file <none>
I20240506 11:38:31.283375 11123 fs_manager.cc:359] Metadata directory not provided
I20240506 11:38:31.283491 11123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:38:31.283795 11123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:38:31.286278 11123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ed72b3a2b6424fa38d5eeefacfe79282"
format_stamp: "Formatted at 2024-05-06 11:38:31 on dist-test-slave-7l0g"
I20240506 11:38:31.287034 11123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ed72b3a2b6424fa38d5eeefacfe79282"
format_stamp: "Formatted at 2024-05-06 11:38:31 on dist-test-slave-7l0g"
I20240506 11:38:31.291839 11123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240506 11:38:31.295015 11137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:31.296363 11123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240506 11:38:31.296519 11123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ed72b3a2b6424fa38d5eeefacfe79282"
format_stamp: "Formatted at 2024-05-06 11:38:31 on dist-test-slave-7l0g"
I20240506 11:38:31.296684 11123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:38:31.326808 11123 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:38:31.327657 11123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:38:31.327927 11123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:38:31.345293 11123 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:45417
I20240506 11:38:31.345325 11188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:45417 every 8 connection(s)
I20240506 11:38:31.346599 11123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:38:31.350065 11189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:38:31.356758 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 11123
I20240506 11:38:31.356942 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:38:31.358783 11189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282: Bootstrap starting.
I20240506 11:38:31.361979 11189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282: Neither blocks nor log segments found. Creating new log.
I20240506 11:38:31.363030 11189 log.cc:824] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282: Log is configured to *not* fsync() on all Append() calls
I20240506 11:38:31.365341 11189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282: No bootstrap required, opened a new log
I20240506 11:38:31.372045 11189 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed72b3a2b6424fa38d5eeefacfe79282" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45417 } }
I20240506 11:38:31.372360 11189 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:38:31.372457 11189 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed72b3a2b6424fa38d5eeefacfe79282, State: Initialized, Role: FOLLOWER
I20240506 11:38:31.372980 11189 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed72b3a2b6424fa38d5eeefacfe79282" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45417 } }
I20240506 11:38:31.373183 11189 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:38:31.373309 11189 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:38:31.373492 11189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:38:31.375486 11189 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed72b3a2b6424fa38d5eeefacfe79282" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45417 } }
I20240506 11:38:31.375952 11189 leader_election.cc:302] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed72b3a2b6424fa38d5eeefacfe79282; no voters:
I20240506 11:38:31.376585 11189 leader_election.cc:288] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:38:31.376745 11194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:38:31.377352 11194 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [term 1 LEADER]: Becoming Leader. State: Replica: ed72b3a2b6424fa38d5eeefacfe79282, State: Running, Role: LEADER
I20240506 11:38:31.377861 11194 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed72b3a2b6424fa38d5eeefacfe79282" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45417 } }
I20240506 11:38:31.378414 11189 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:38:31.380627 11196 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed72b3a2b6424fa38d5eeefacfe79282. Latest consensus state: current_term: 1 leader_uuid: "ed72b3a2b6424fa38d5eeefacfe79282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed72b3a2b6424fa38d5eeefacfe79282" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45417 } } }
I20240506 11:38:31.380982 11196 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [sys.catalog]: This master's current role is: LEADER
I20240506 11:38:31.383261 11202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:38:31.383164 11195 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed72b3a2b6424fa38d5eeefacfe79282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed72b3a2b6424fa38d5eeefacfe79282" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45417 } } }
I20240506 11:38:31.383415 11195 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282 [sys.catalog]: This master's current role is: LEADER
I20240506 11:38:31.388303 11202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:38:31.396723 11202 catalog_manager.cc:1347] Generated new cluster ID: c714d28dc1d047b2b2d33313b315153a
I20240506 11:38:31.396850 11202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:38:31.419232 11202 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:38:31.420698 11202 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:38:31.434155 11202 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ed72b3a2b6424fa38d5eeefacfe79282: Generated new TSK 0
I20240506 11:38:31.434964 11202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:38:31.442806 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45417
--builtin_ntp_servers=127.10.45.148:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20240506 11:38:31.756390 11213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:38:31.756675 11213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:38:31.756816 11213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:38:31.764318 11213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:38:31.764519 11213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:38:31.773443 11213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:38:31.774755 11213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:38:31.776437 11213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:38:31.788686 11218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:31.789561 11221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:31.790380 11219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:38:31.791307 11213 server_base.cc:1034] running on GCE node
I20240506 11:38:31.791888 11213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:38:31.792757 11213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:38:31.794034 11213 hybrid_clock.cc:648] HybridClock initialized: now 1714995511793969 us; error 46 us; skew 500 ppm
I20240506 11:38:31.794415 11213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:38:31.796751 11213 webserver.cc:460] Webserver started at http://127.10.45.129:44195/ using document root <none> and password file <none>
I20240506 11:38:31.797459 11213 fs_manager.cc:359] Metadata directory not provided
I20240506 11:38:31.797587 11213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:38:31.797880 11213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:38:31.800428 11213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d"
format_stamp: "Formatted at 2024-05-06 11:38:31 on dist-test-slave-7l0g"
I20240506 11:38:31.801193 11213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d"
format_stamp: "Formatted at 2024-05-06 11:38:31 on dist-test-slave-7l0g"
I20240506 11:38:31.806182 11213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240506 11:38:31.809410 11227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:31.810664 11213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240506 11:38:31.810813 11213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d"
format_stamp: "Formatted at 2024-05-06 11:38:31 on dist-test-slave-7l0g"
I20240506 11:38:31.810994 11213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:38:31.841002 11213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:38:31.841822 11213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:38:31.842101 11213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:38:31.843071 11213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:38:31.844502 11213 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:38:31.844594 11213 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:31.844712 11213 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:38:31.844764 11213 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:31.874538 11213 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:37939
I20240506 11:38:31.874601 11339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:37939 every 8 connection(s)
I20240506 11:38:31.875792 11213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:38:31.881836 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 11213
I20240506 11:38:31.882011 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:38:31.885244 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45417
--builtin_ntp_servers=127.10.45.148:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20240506 11:38:31.888662 11340 heartbeater.cc:347] Connected to a master server at 127.10.45.190:45417
I20240506 11:38:31.888948 11340 heartbeater.cc:464] Registering TS with master...
I20240506 11:38:31.889973 11340 heartbeater.cc:511] Master 127.10.45.190:45417 requested a full tablet report, sending...
I20240506 11:38:31.892055 11154 ts_manager.cc:194] Registered new tserver with Master: b3b2915ef58149b1a8f1961a1d3f7a0d (127.10.45.129:37939)
I20240506 11:38:31.893435 11154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:55107
W20240506 11:38:32.187543 11344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:38:32.187783 11344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:38:32.187902 11344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:38:32.194537 11344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:38:32.194711 11344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:38:32.203164 11344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:38:32.204388 11344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:38:32.205914 11344 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:38:32.216974 11349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:32.217800 11350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:32.218667 11352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:38:32.219516 11344 server_base.cc:1034] running on GCE node
I20240506 11:38:32.220098 11344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:38:32.220959 11344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:38:32.222190 11344 hybrid_clock.cc:648] HybridClock initialized: now 1714995512222140 us; error 51 us; skew 500 ppm
I20240506 11:38:32.222441 11344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:38:32.224525 11344 webserver.cc:460] Webserver started at http://127.10.45.130:38729/ using document root <none> and password file <none>
I20240506 11:38:32.225211 11344 fs_manager.cc:359] Metadata directory not provided
I20240506 11:38:32.225335 11344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:38:32.225621 11344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:38:32.228214 11344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41"
format_stamp: "Formatted at 2024-05-06 11:38:32 on dist-test-slave-7l0g"
I20240506 11:38:32.228997 11344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41"
format_stamp: "Formatted at 2024-05-06 11:38:32 on dist-test-slave-7l0g"
I20240506 11:38:32.233932 11344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240506 11:38:32.237166 11358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:32.238343 11344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240506 11:38:32.238487 11344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41"
format_stamp: "Formatted at 2024-05-06 11:38:32 on dist-test-slave-7l0g"
I20240506 11:38:32.238641 11344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:38:32.262570 11344 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:38:32.263391 11344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:38:32.263674 11344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:38:32.264559 11344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:38:32.265894 11344 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:38:32.265970 11344 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:32.266098 11344 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:38:32.266151 11344 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:32.294903 11344 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:33477
I20240506 11:38:32.294936 11470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:33477 every 8 connection(s)
I20240506 11:38:32.296150 11344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:38:32.302359 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 11344
I20240506 11:38:32.302512 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:38:32.305238 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45417
--builtin_ntp_servers=127.10.45.148:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20240506 11:38:32.309279 11471 heartbeater.cc:347] Connected to a master server at 127.10.45.190:45417
I20240506 11:38:32.309526 11471 heartbeater.cc:464] Registering TS with master...
I20240506 11:38:32.310212 11471 heartbeater.cc:511] Master 127.10.45.190:45417 requested a full tablet report, sending...
I20240506 11:38:32.311491 11154 ts_manager.cc:194] Registered new tserver with Master: ddf64df9c60a4bcdb63ef4d303d8bc41 (127.10.45.130:33477)
I20240506 11:38:32.312191 11154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:40061
W20240506 11:38:32.610073 11474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:38:32.610416 11474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:38:32.610510 11474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:38:32.617997 11474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:38:32.618178 11474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:38:32.627094 11474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:38:32.628333 11474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:38:32.629951 11474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:38:32.641450 11479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:32.642393 11480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:32.643146 11482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:38:32.643939 11474 server_base.cc:1034] running on GCE node
I20240506 11:38:32.644495 11474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:38:32.645331 11474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:38:32.646541 11474 hybrid_clock.cc:648] HybridClock initialized: now 1714995512646493 us; error 35 us; skew 500 ppm
I20240506 11:38:32.646783 11474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:38:32.652514 11474 webserver.cc:460] Webserver started at http://127.10.45.131:33273/ using document root <none> and password file <none>
I20240506 11:38:32.653180 11474 fs_manager.cc:359] Metadata directory not provided
I20240506 11:38:32.653290 11474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:38:32.653527 11474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:38:32.655941 11474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1770f9742fb945faaff25dbdf1bcfde2"
format_stamp: "Formatted at 2024-05-06 11:38:32 on dist-test-slave-7l0g"
I20240506 11:38:32.656590 11474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1770f9742fb945faaff25dbdf1bcfde2"
format_stamp: "Formatted at 2024-05-06 11:38:32 on dist-test-slave-7l0g"
I20240506 11:38:32.661051 11474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240506 11:38:32.663955 11488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:32.665122 11474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240506 11:38:32.665279 11474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1770f9742fb945faaff25dbdf1bcfde2"
format_stamp: "Formatted at 2024-05-06 11:38:32 on dist-test-slave-7l0g"
I20240506 11:38:32.665441 11474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:38:32.692121 11474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:38:32.693198 11474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:38:32.693511 11474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:38:32.694733 11474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:38:32.696905 11474 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:38:32.697013 11474 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:32.697151 11474 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:38:32.697224 11474 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:32.730283 11474 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:37727
I20240506 11:38:32.730304 11600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:37727 every 8 connection(s)
I20240506 11:38:32.732110 11474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:38:32.741762 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 11474
I20240506 11:38:32.741921 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:38:32.746026 11601 heartbeater.cc:347] Connected to a master server at 127.10.45.190:45417
I20240506 11:38:32.746295 11601 heartbeater.cc:464] Registering TS with master...
I20240506 11:38:32.747042 11601 heartbeater.cc:511] Master 127.10.45.190:45417 requested a full tablet report, sending...
I20240506 11:38:32.748389 11154 ts_manager.cc:194] Registered new tserver with Master: 1770f9742fb945faaff25dbdf1bcfde2 (127.10.45.131:37727)
I20240506 11:38:32.749631 11154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:41757
I20240506 11:38:32.757234 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:38:32.771975 11154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34500:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20240506 11:38:32.776954 11154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 11:38:32.817613 11406 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae0a391b65d403780734d103a0ff809 (DEFAULT_TABLE table=TestTable [id=740cd0afe91e4ef5a6c60fcfe4903135]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:38:32.819243 11406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae0a391b65d403780734d103a0ff809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:38:32.820114 11275 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae0a391b65d403780734d103a0ff809 (DEFAULT_TABLE table=TestTable [id=740cd0afe91e4ef5a6c60fcfe4903135]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:38:32.821331 11275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae0a391b65d403780734d103a0ff809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:38:32.821390 11536 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae0a391b65d403780734d103a0ff809 (DEFAULT_TABLE table=TestTable [id=740cd0afe91e4ef5a6c60fcfe4903135]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:38:32.822695 11536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ae0a391b65d403780734d103a0ff809. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:38:32.838708 11619 tablet_bootstrap.cc:492] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Bootstrap starting.
I20240506 11:38:32.841709 11620 tablet_bootstrap.cc:492] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d: Bootstrap starting.
I20240506 11:38:32.845494 11620 tablet_bootstrap.cc:654] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d: Neither blocks nor log segments found. Creating new log.
I20240506 11:38:32.845507 11619 tablet_bootstrap.cc:654] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Neither blocks nor log segments found. Creating new log.
I20240506 11:38:32.846756 11620 log.cc:824] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d: Log is configured to *not* fsync() on all Append() calls
I20240506 11:38:32.846758 11619 log.cc:824] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Log is configured to *not* fsync() on all Append() calls
I20240506 11:38:32.848021 11621 tablet_bootstrap.cc:492] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2: Bootstrap starting.
I20240506 11:38:32.851646 11621 tablet_bootstrap.cc:654] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2: Neither blocks nor log segments found. Creating new log.
I20240506 11:38:32.852821 11621 log.cc:824] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2: Log is configured to *not* fsync() on all Append() calls
I20240506 11:38:32.853330 11619 tablet_bootstrap.cc:492] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: No bootstrap required, opened a new log
I20240506 11:38:32.853717 11619 ts_tablet_manager.cc:1367] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240506 11:38:32.855408 11620 tablet_bootstrap.cc:492] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d: No bootstrap required, opened a new log
I20240506 11:38:32.855690 11620 ts_tablet_manager.cc:1367] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20240506 11:38:32.865518 11619 raft_consensus.cc:348] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:32.866811 11619 raft_consensus.cc:374] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:38:32.867002 11619 raft_consensus.cc:729] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf64df9c60a4bcdb63ef4d303d8bc41, State: Initialized, Role: FOLLOWER
I20240506 11:38:32.866832 11620 raft_consensus.cc:348] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:32.867338 11620 raft_consensus.cc:374] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:38:32.867466 11620 raft_consensus.cc:729] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b2915ef58149b1a8f1961a1d3f7a0d, State: Initialized, Role: FOLLOWER
I20240506 11:38:32.867928 11619 consensus_queue.cc:260] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:32.868243 11620 consensus_queue.cc:260] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:32.868465 11621 tablet_bootstrap.cc:492] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2: No bootstrap required, opened a new log
I20240506 11:38:32.868824 11621 ts_tablet_manager.cc:1367] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.014s
I20240506 11:38:32.877882 11621 raft_consensus.cc:348] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:32.878348 11621 raft_consensus.cc:374] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:38:32.878491 11621 raft_consensus.cc:729] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1770f9742fb945faaff25dbdf1bcfde2, State: Initialized, Role: FOLLOWER
I20240506 11:38:32.879349 11621 consensus_queue.cc:260] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:32.892501 11340 heartbeater.cc:503] Master 127.10.45.190:45417 was elected leader, sending a full tablet report...
I20240506 11:38:32.896031 11601 heartbeater.cc:503] Master 127.10.45.190:45417 was elected leader, sending a full tablet report...
I20240506 11:38:32.896838 11621 ts_tablet_manager.cc:1398] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2: Time spent starting tablet: real 0.028s user 0.017s sys 0.009s
I20240506 11:38:32.898082 11619 ts_tablet_manager.cc:1398] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Time spent starting tablet: real 0.044s user 0.020s sys 0.000s
I20240506 11:38:32.898494 11471 heartbeater.cc:503] Master 127.10.45.190:45417 was elected leader, sending a full tablet report...
I20240506 11:38:32.900007 11625 raft_consensus.cc:482] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:38:32.902036 11620 ts_tablet_manager.cc:1398] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d: Time spent starting tablet: real 0.046s user 0.017s sys 0.018s
I20240506 11:38:32.904182 11625 raft_consensus.cc:504] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:32.906412 11625 leader_election.cc:288] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1770f9742fb945faaff25dbdf1bcfde2 (127.10.45.131:37727), b3b2915ef58149b1a8f1961a1d3f7a0d (127.10.45.129:37939)
I20240506 11:38:32.914795 11295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae0a391b65d403780734d103a0ff809" candidate_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" is_pre_election: true
I20240506 11:38:32.914795 11556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae0a391b65d403780734d103a0ff809" candidate_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1770f9742fb945faaff25dbdf1bcfde2" is_pre_election: true
I20240506 11:38:32.915338 11295 raft_consensus.cc:2454] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddf64df9c60a4bcdb63ef4d303d8bc41 in term 0.
I20240506 11:38:32.915338 11556 raft_consensus.cc:2454] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddf64df9c60a4bcdb63ef4d303d8bc41 in term 0.
I20240506 11:38:32.915989 11361 leader_election.cc:302] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1770f9742fb945faaff25dbdf1bcfde2, ddf64df9c60a4bcdb63ef4d303d8bc41; no voters:
I20240506 11:38:32.916330 11625 raft_consensus.cc:2781] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 11:38:32.916471 11625 raft_consensus.cc:482] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:38:32.916581 11625 raft_consensus.cc:3037] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:38:32.918622 11625 raft_consensus.cc:504] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:32.919255 11625 leader_election.cc:288] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [CANDIDATE]: Term 1 election: Requested vote from peers 1770f9742fb945faaff25dbdf1bcfde2 (127.10.45.131:37727), b3b2915ef58149b1a8f1961a1d3f7a0d (127.10.45.129:37939)
I20240506 11:38:32.919621 11556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae0a391b65d403780734d103a0ff809" candidate_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1770f9742fb945faaff25dbdf1bcfde2"
I20240506 11:38:32.919790 11295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae0a391b65d403780734d103a0ff809" candidate_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d"
I20240506 11:38:32.919826 11556 raft_consensus.cc:3037] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:38:32.919984 11295 raft_consensus.cc:3037] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:38:32.922860 11295 raft_consensus.cc:2454] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddf64df9c60a4bcdb63ef4d303d8bc41 in term 1.
I20240506 11:38:32.922860 11556 raft_consensus.cc:2454] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddf64df9c60a4bcdb63ef4d303d8bc41 in term 1.
I20240506 11:38:32.923327 11359 leader_election.cc:302] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b2915ef58149b1a8f1961a1d3f7a0d, ddf64df9c60a4bcdb63ef4d303d8bc41; no voters:
I20240506 11:38:32.923687 11625 raft_consensus.cc:2781] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:38:32.923820 11625 raft_consensus.cc:686] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 1 LEADER]: Becoming Leader. State: Replica: ddf64df9c60a4bcdb63ef4d303d8bc41, State: Running, Role: LEADER
I20240506 11:38:32.924259 11625 consensus_queue.cc:237] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:32.928932 11152 catalog_manager.cc:5484] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddf64df9c60a4bcdb63ef4d303d8bc41 (127.10.45.130). New cstate: current_term: 1 leader_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } health_report { overall_health: UNKNOWN } } }
I20240506 11:38:32.962007 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:38:32.964560 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b3b2915ef58149b1a8f1961a1d3f7a0d to finish bootstrapping
I20240506 11:38:32.972182 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ddf64df9c60a4bcdb63ef4d303d8bc41 to finish bootstrapping
I20240506 11:38:32.978626 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1770f9742fb945faaff25dbdf1bcfde2 to finish bootstrapping
W20240506 11:38:32.983899 11602 tablet.cc:2369] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:38:33.010919 11556 raft_consensus.cc:1261] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 1 FOLLOWER]: Refusing update from remote peer ddf64df9c60a4bcdb63ef4d303d8bc41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:38:33.010919 11295 raft_consensus.cc:1261] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [term 1 FOLLOWER]: Refusing update from remote peer ddf64df9c60a4bcdb63ef4d303d8bc41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:38:33.011848 11628 consensus_queue.cc:1035] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:38:33.012136 11625 consensus_queue.cc:1035] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:38:33.026574 11634 mvcc.cc:205] Tried to move back new op lower bound from 7024621621281914880 to 7024621620942000128. Current Snapshot: MvccSnapshot[applied={T|T < 7024621621281914880}]
I20240506 11:38:33.027761 11636 mvcc.cc:205] Tried to move back new op lower bound from 7024621621281914880 to 7024621620942000128. Current Snapshot: MvccSnapshot[applied={T|T < 7024621621281914880}]
I20240506 11:38:33.163959 11536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:38:33.163962 11275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:38:33.166157 11406 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ed72b3a2b6424fa38d5eeefacfe79282 | 127.10.45.190:45417 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:40071 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
1770f9742fb945faaff25dbdf1bcfde2 | 127.10.45.131:37727 | HEALTHY | <none> | 0 | 0
b3b2915ef58149b1a8f1961a1d3f7a0d | 127.10.45.129:37939 | HEALTHY | <none> | 0 | 0
ddf64df9c60a4bcdb63ef4d303d8bc41 | 127.10.45.130:33477 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.45.129 | experimental | 127.10.45.129:37939
local_ip_for_outbound_sockets | 127.10.45.130 | experimental | 127.10.45.130:33477
local_ip_for_outbound_sockets | 127.10.45.131 | experimental | 127.10.45.131:37727
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.45.129:37939
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.45.130:33477
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.45.131:37727
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:40071 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240506 11:38:33.208195 10422 log_verifier.cc:126] Checking tablet 3ae0a391b65d403780734d103a0ff809
I20240506 11:38:33.241487 10422 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3ae0a391b65d403780734d103a0ff809
I20240506 11:38:33.682690 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 11344
I20240506 11:38:33.683461 11466 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:38:33.777278 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 11344
I20240506 11:38:33.795526 11556 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ae0a391b65d403780734d103a0ff809"
dest_uuid: "1770f9742fb945faaff25dbdf1bcfde2"
from {username='slave'} at 127.0.0.1:53928
I20240506 11:38:33.795820 11556 raft_consensus.cc:482] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:38:33.795964 11556 raft_consensus.cc:3037] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:38:33.797732 11556 raft_consensus.cc:504] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:33.799371 11556 leader_election.cc:288] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [CANDIDATE]: Term 2 election: Requested vote from peers ddf64df9c60a4bcdb63ef4d303d8bc41 (127.10.45.130:33477), b3b2915ef58149b1a8f1961a1d3f7a0d (127.10.45.129:37939)
I20240506 11:38:33.810175 11294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae0a391b65d403780734d103a0ff809" candidate_uuid: "1770f9742fb945faaff25dbdf1bcfde2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d"
I20240506 11:38:33.810494 11294 raft_consensus.cc:3037] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:38:33.814414 11294 raft_consensus.cc:2454] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1770f9742fb945faaff25dbdf1bcfde2 in term 2.
W20240506 11:38:33.814775 11492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.45.130:33477: connect: Connection refused (error 111)
I20240506 11:38:33.815032 11489 leader_election.cc:302] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1770f9742fb945faaff25dbdf1bcfde2, b3b2915ef58149b1a8f1961a1d3f7a0d; no voters:
W20240506 11:38:33.819622 11607 meta_cache.cc:302] tablet 3ae0a391b65d403780734d103a0ff809: replica ddf64df9c60a4bcdb63ef4d303d8bc41 (127.10.45.130:33477) has failed: Network error: Client connection negotiation failed: client connection to 127.10.45.130:33477: connect: Connection refused (error 111)
I20240506 11:38:33.820350 11681 raft_consensus.cc:2781] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 11:38:33.820839 11681 raft_consensus.cc:686] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [term 2 LEADER]: Becoming Leader. State: Replica: 1770f9742fb945faaff25dbdf1bcfde2, State: Running, Role: LEADER
I20240506 11:38:33.821383 11681 consensus_queue.cc:237] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
W20240506 11:38:33.822548 11492 leader_election.cc:334] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ddf64df9c60a4bcdb63ef4d303d8bc41 (127.10.45.130:33477): Network error: Client connection negotiation failed: client connection to 127.10.45.130:33477: connect: Connection refused (error 111)
I20240506 11:38:33.824680 11151 catalog_manager.cc:5484] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 reported cstate change: term changed from 1 to 2, leader changed from ddf64df9c60a4bcdb63ef4d303d8bc41 (127.10.45.130) to 1770f9742fb945faaff25dbdf1bcfde2 (127.10.45.131). New cstate: current_term: 2 leader_uuid: "1770f9742fb945faaff25dbdf1bcfde2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } health_report { overall_health: UNKNOWN } } }
I20240506 11:38:33.836027 11294 raft_consensus.cc:1261] T 3ae0a391b65d403780734d103a0ff809 P b3b2915ef58149b1a8f1961a1d3f7a0d [term 2 FOLLOWER]: Refusing update from remote peer 1770f9742fb945faaff25dbdf1bcfde2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240506 11:38:33.836606 11681 consensus_queue.cc:1035] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240506 11:38:33.836843 11492 consensus_peers.cc:485] T 3ae0a391b65d403780734d103a0ff809 P 1770f9742fb945faaff25dbdf1bcfde2 -> Peer ddf64df9c60a4bcdb63ef4d303d8bc41 (127.10.45.130:33477): Couldn't send request to peer ddf64df9c60a4bcdb63ef4d303d8bc41. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.130:33477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:38:33.905652 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:33477
--local_ip_for_outbound_sockets=127.10.45.130
--tserver_master_addrs=127.10.45.190:45417
--webserver_port=38729
--webserver_interface=127.10.45.130
--builtin_ntp_servers=127.10.45.148:40071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20240506 11:38:34.211146 11693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:38:34.211416 11693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:38:34.211547 11693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:38:34.218274 11693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:38:34.218456 11693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:38:34.226696 11693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:33477
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=38729
--tserver_master_addrs=127.10.45.190:45417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:38:34.227918 11693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:38:34.229507 11693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:38:34.240983 11699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:34.241830 11701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:34.242655 11698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:38:34.243451 11693 server_base.cc:1034] running on GCE node
I20240506 11:38:34.243970 11693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:38:34.244843 11693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:38:34.246065 11693 hybrid_clock.cc:648] HybridClock initialized: now 1714995514245991 us; error 57 us; skew 500 ppm
I20240506 11:38:34.246332 11693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:38:34.248432 11693 webserver.cc:460] Webserver started at http://127.10.45.130:38729/ using document root <none> and password file <none>
I20240506 11:38:34.249109 11693 fs_manager.cc:359] Metadata directory not provided
I20240506 11:38:34.249243 11693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:38:34.253957 11693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240506 11:38:34.258883 11707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:34.260177 11693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240506 11:38:34.260321 11693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41"
format_stamp: "Formatted at 2024-05-06 11:38:32 on dist-test-slave-7l0g"
I20240506 11:38:34.261080 11693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:38:34.286300 11693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:38:34.287212 11693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:38:34.287493 11693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:38:34.288398 11693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:38:34.290210 11714 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240506 11:38:34.295361 11693 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 11:38:34.295471 11693 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20240506 11:38:34.295588 11693 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 11:38:34.298178 11693 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 11:38:34.298254 11693 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240506 11:38:34.298555 11714 tablet_bootstrap.cc:492] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Bootstrap starting.
I20240506 11:38:34.316979 11714 log.cc:824] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Log is configured to *not* fsync() on all Append() calls
I20240506 11:38:34.334550 11693 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:33477
I20240506 11:38:34.334568 11821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:33477 every 8 connection(s)
I20240506 11:38:34.336048 11693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:38:34.343175 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 11693
I20240506 11:38:34.352098 11822 heartbeater.cc:347] Connected to a master server at 127.10.45.190:45417
I20240506 11:38:34.352414 11822 heartbeater.cc:464] Registering TS with master...
I20240506 11:38:34.353395 11822 heartbeater.cc:511] Master 127.10.45.190:45417 requested a full tablet report, sending...
I20240506 11:38:34.356094 11151 ts_manager.cc:194] Re-registered known tserver with Master: ddf64df9c60a4bcdb63ef4d303d8bc41 (127.10.45.130:33477)
I20240506 11:38:34.356575 11714 tablet_bootstrap.cc:492] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 11:38:34.357265 11714 tablet_bootstrap.cc:492] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Bootstrap complete.
I20240506 11:38:34.357486 11151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:45677
I20240506 11:38:34.359117 11714 ts_tablet_manager.cc:1367] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Time spent bootstrapping tablet: real 0.061s user 0.046s sys 0.015s
I20240506 11:38:34.365346 11714 raft_consensus.cc:348] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:34.366231 11714 raft_consensus.cc:729] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf64df9c60a4bcdb63ef4d303d8bc41, State: Initialized, Role: FOLLOWER
I20240506 11:38:34.366899 11714 consensus_queue.cc:260] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1770f9742fb945faaff25dbdf1bcfde2" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 37727 } } peers { permanent_uuid: "ddf64df9c60a4bcdb63ef4d303d8bc41" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 33477 } } peers { permanent_uuid: "b3b2915ef58149b1a8f1961a1d3f7a0d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37939 } }
I20240506 11:38:34.367936 11822 heartbeater.cc:503] Master 127.10.45.190:45417 was elected leader, sending a full tablet report...
I20240506 11:38:34.368395 11714 ts_tablet_manager.cc:1398] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240506 11:38:34.412102 11777 raft_consensus.cc:3037] T 3ae0a391b65d403780734d103a0ff809 P ddf64df9c60a4bcdb63ef4d303d8bc41 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:38:34.436626 11848 mvcc.cc:205] Tried to move back new op lower bound from 7024621624925044736 to 7024621624616996864. Current Snapshot: MvccSnapshot[applied={T|T < 7024621624859811840}]
I20240506 11:38:34.444734 11536 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240506 11:38:34.455209 11275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:38:34.457499 11756 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ed72b3a2b6424fa38d5eeefacfe79282 | 127.10.45.190:45417 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:40071 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
1770f9742fb945faaff25dbdf1bcfde2 | 127.10.45.131:37727 | HEALTHY | <none> | 1 | 0
b3b2915ef58149b1a8f1961a1d3f7a0d | 127.10.45.129:37939 | HEALTHY | <none> | 0 | 0
ddf64df9c60a4bcdb63ef4d303d8bc41 | 127.10.45.130:33477 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.45.129 | experimental | 127.10.45.129:37939
local_ip_for_outbound_sockets | 127.10.45.130 | experimental | 127.10.45.130:33477
local_ip_for_outbound_sockets | 127.10.45.131 | experimental | 127.10.45.131:37727
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.45.129:37939
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.45.130:33477
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.45.131:37727
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:40071 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240506 11:38:34.496059 10422 log_verifier.cc:126] Checking tablet 3ae0a391b65d403780734d103a0ff809
I20240506 11:38:34.530273 10422 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 3ae0a391b65d403780734d103a0ff809
I20240506 11:38:34.859800 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 11213
I20240506 11:38:34.860774 11335 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:38:34.966244 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 11213
I20240506 11:38:34.986063 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 11693
I20240506 11:38:34.991483 11817 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:38:35.072324 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 11693
I20240506 11:38:35.087801 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 11474
I20240506 11:38:35.088716 11596 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:38:35.173280 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 11474
I20240506 11:38:35.189463 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 11123
I20240506 11:38:35.190496 11184 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:38:35.290688 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 11123
2024-05-06T11:38:35Z chronyd exiting
[ OK ] RaftConsensusElectionITest.RunLeaderElection (4424 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElections
I20240506 11:38:35.332522 10422 test_util.cc:268] Using random seed: -2105829487
I20240506 11:38:35.334170 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:38:35.334229 10422 ts_itest-base.cc:113] --------------
I20240506 11:38:35.334286 10422 ts_itest-base.cc:114] 3 tablet servers
I20240506 11:38:35.334332 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:38:35.334388 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:38:35Z Disabled control of system clock
I20240506 11:38:35.356961 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:37311
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:37311 with env {}
W20240506 11:38:35.649519 11880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:38:35.649818 11880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:38:35.649945 11880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:38:35.656687 11880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:38:35.656785 11880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:38:35.656880 11880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:38:35.656956 11880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:38:35.664882 11880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:37311
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:37311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:38:35.666052 11880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:38:35.667609 11880 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:38:35.678347 11885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:35.679189 11886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:35.679967 11888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:38:35.680810 11880 server_base.cc:1034] running on GCE node
I20240506 11:38:35.681450 11880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:38:35.682754 11880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:38:35.683979 11880 hybrid_clock.cc:648] HybridClock initialized: now 1714995515683917 us; error 36 us; skew 500 ppm
I20240506 11:38:35.684270 11880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:38:35.686211 11880 webserver.cc:460] Webserver started at http://127.10.45.190:36485/ using document root <none> and password file <none>
I20240506 11:38:35.686816 11880 fs_manager.cc:359] Metadata directory not provided
I20240506 11:38:35.686933 11880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:38:35.687233 11880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:38:35.689509 11880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e5939149a922434d86d9213ce9d3501d"
format_stamp: "Formatted at 2024-05-06 11:38:35 on dist-test-slave-7l0g"
I20240506 11:38:35.690253 11880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e5939149a922434d86d9213ce9d3501d"
format_stamp: "Formatted at 2024-05-06 11:38:35 on dist-test-slave-7l0g"
I20240506 11:38:35.695003 11880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20240506 11:38:35.697947 11894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:35.699117 11880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20240506 11:38:35.699261 11880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e5939149a922434d86d9213ce9d3501d"
format_stamp: "Formatted at 2024-05-06 11:38:35 on dist-test-slave-7l0g"
I20240506 11:38:35.699407 11880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:38:35.727972 11880 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:38:35.728686 11880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:38:35.728928 11880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:38:35.745105 11880 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:37311
I20240506 11:38:35.745142 11945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:37311 every 8 connection(s)
I20240506 11:38:35.746279 11880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:38:35.749408 11946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:38:35.751098 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 11880
I20240506 11:38:35.751256 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:38:35.758584 11946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d: Bootstrap starting.
I20240506 11:38:35.761615 11946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d: Neither blocks nor log segments found. Creating new log.
I20240506 11:38:35.762918 11946 log.cc:824] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d: Log is configured to *not* fsync() on all Append() calls
I20240506 11:38:35.765295 11946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d: No bootstrap required, opened a new log
I20240506 11:38:35.771502 11946 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5939149a922434d86d9213ce9d3501d" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37311 } }
I20240506 11:38:35.771791 11946 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:38:35.771876 11946 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5939149a922434d86d9213ce9d3501d, State: Initialized, Role: FOLLOWER
I20240506 11:38:35.772480 11946 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5939149a922434d86d9213ce9d3501d" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37311 } }
I20240506 11:38:35.772665 11946 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:38:35.772781 11946 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:38:35.772917 11946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:38:35.774757 11946 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5939149a922434d86d9213ce9d3501d" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37311 } }
I20240506 11:38:35.775194 11946 leader_election.cc:302] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5939149a922434d86d9213ce9d3501d; no voters:
I20240506 11:38:35.775687 11946 leader_election.cc:288] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:38:35.775851 11951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:38:35.776401 11951 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [term 1 LEADER]: Becoming Leader. State: Replica: e5939149a922434d86d9213ce9d3501d, State: Running, Role: LEADER
I20240506 11:38:35.776880 11951 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5939149a922434d86d9213ce9d3501d" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37311 } }
I20240506 11:38:35.777398 11946 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:38:35.779781 11953 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5939149a922434d86d9213ce9d3501d. Latest consensus state: current_term: 1 leader_uuid: "e5939149a922434d86d9213ce9d3501d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5939149a922434d86d9213ce9d3501d" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37311 } } }
I20240506 11:38:35.779865 11952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5939149a922434d86d9213ce9d3501d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5939149a922434d86d9213ce9d3501d" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37311 } } }
I20240506 11:38:35.780476 11953 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [sys.catalog]: This master's current role is: LEADER
I20240506 11:38:35.780484 11952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d [sys.catalog]: This master's current role is: LEADER
I20240506 11:38:35.781800 11960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:38:35.786656 11960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:38:35.794340 11960 catalog_manager.cc:1347] Generated new cluster ID: 373b73c14028427d8326899a10910347
I20240506 11:38:35.794436 11960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:38:35.818387 11960 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:38:35.819681 11960 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:38:35.834120 11960 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e5939149a922434d86d9213ce9d3501d: Generated new TSK 0
I20240506 11:38:35.835173 11960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:38:35.848678 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:37311
--builtin_ntp_servers=127.10.45.148:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20240506 11:38:36.154417 11970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:38:36.154685 11970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:38:36.154844 11970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:38:36.161777 11970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:38:36.161985 11970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:38:36.170336 11970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.45.190:37311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:38:36.171604 11970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:38:36.173235 11970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:38:36.185163 11975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:36.185973 11976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:36.186775 11978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:38:36.187721 11970 server_base.cc:1034] running on GCE node
I20240506 11:38:36.188295 11970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:38:36.189169 11970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:38:36.190383 11970 hybrid_clock.cc:648] HybridClock initialized: now 1714995516190326 us; error 30 us; skew 500 ppm
I20240506 11:38:36.190693 11970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:38:36.192844 11970 webserver.cc:460] Webserver started at http://127.10.45.129:33677/ using document root <none> and password file <none>
I20240506 11:38:36.193521 11970 fs_manager.cc:359] Metadata directory not provided
I20240506 11:38:36.193641 11970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:38:36.193955 11970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:38:36.196553 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e"
format_stamp: "Formatted at 2024-05-06 11:38:36 on dist-test-slave-7l0g"
I20240506 11:38:36.197273 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e"
format_stamp: "Formatted at 2024-05-06 11:38:36 on dist-test-slave-7l0g"
I20240506 11:38:36.201925 11970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240506 11:38:36.205042 11984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:36.206342 11970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240506 11:38:36.206490 11970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e"
format_stamp: "Formatted at 2024-05-06 11:38:36 on dist-test-slave-7l0g"
I20240506 11:38:36.206657 11970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:38:36.232132 11970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:38:36.232903 11970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:38:36.233193 11970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:38:36.234032 11970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:38:36.235364 11970 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:38:36.235431 11970 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:36.235517 11970 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:38:36.235558 11970 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:36.264364 11970 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:39317
I20240506 11:38:36.264430 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:39317 every 8 connection(s)
I20240506 11:38:36.265550 11970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:38:36.266564 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 11970
I20240506 11:38:36.266765 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:38:36.270121 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:37311
--builtin_ntp_servers=127.10.45.148:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20240506 11:38:36.278147 12097 heartbeater.cc:347] Connected to a master server at 127.10.45.190:37311
I20240506 11:38:36.278396 12097 heartbeater.cc:464] Registering TS with master...
I20240506 11:38:36.279192 12097 heartbeater.cc:511] Master 127.10.45.190:37311 requested a full tablet report, sending...
I20240506 11:38:36.281277 11911 ts_manager.cc:194] Registered new tserver with Master: b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:36.283548 11911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:47259
W20240506 11:38:36.572568 12101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:38:36.572829 12101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:38:36.572918 12101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:38:36.580091 12101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:38:36.580263 12101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:38:36.588975 12101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.45.190:37311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:38:36.590191 12101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:38:36.591902 12101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:38:36.603739 12106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:36.604828 12107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:36.605549 12109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:38:36.606348 12101 server_base.cc:1034] running on GCE node
I20240506 11:38:36.606981 12101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:38:36.607918 12101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:38:36.609131 12101 hybrid_clock.cc:648] HybridClock initialized: now 1714995516609056 us; error 47 us; skew 500 ppm
I20240506 11:38:36.609405 12101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:38:36.611668 12101 webserver.cc:460] Webserver started at http://127.10.45.130:34893/ using document root <none> and password file <none>
I20240506 11:38:36.612581 12101 fs_manager.cc:359] Metadata directory not provided
I20240506 11:38:36.612732 12101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:38:36.613081 12101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:38:36.616780 12101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f1dfb32eb83944708aa5de1a681ed391"
format_stamp: "Formatted at 2024-05-06 11:38:36 on dist-test-slave-7l0g"
I20240506 11:38:36.617720 12101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f1dfb32eb83944708aa5de1a681ed391"
format_stamp: "Formatted at 2024-05-06 11:38:36 on dist-test-slave-7l0g"
I20240506 11:38:36.622915 12101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240506 11:38:36.626657 12115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:36.628232 12101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240506 11:38:36.628398 12101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f1dfb32eb83944708aa5de1a681ed391"
format_stamp: "Formatted at 2024-05-06 11:38:36 on dist-test-slave-7l0g"
I20240506 11:38:36.628561 12101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:38:36.657330 12101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:38:36.658131 12101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:38:36.658398 12101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:38:36.659256 12101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:38:36.660630 12101 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:38:36.660725 12101 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:36.660859 12101 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:38:36.660902 12101 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:36.691679 12101 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:45689
I20240506 11:38:36.691732 12227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:45689 every 8 connection(s)
I20240506 11:38:36.692910 12101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:38:36.699023 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 12101
I20240506 11:38:36.699216 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:38:36.702596 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:37311
--builtin_ntp_servers=127.10.45.148:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20240506 11:38:36.705705 12228 heartbeater.cc:347] Connected to a master server at 127.10.45.190:37311
I20240506 11:38:36.706032 12228 heartbeater.cc:464] Registering TS with master...
I20240506 11:38:36.707001 12228 heartbeater.cc:511] Master 127.10.45.190:37311 requested a full tablet report, sending...
I20240506 11:38:36.708631 11911 ts_manager.cc:194] Registered new tserver with Master: f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689)
I20240506 11:38:36.709676 11911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:41737
W20240506 11:38:37.007810 12231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:38:37.008075 12231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:38:37.008185 12231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:38:37.015219 12231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:38:37.015381 12231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:38:37.023587 12231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.45.190:37311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:38:37.024760 12231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:38:37.026300 12231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:38:37.037833 12236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:37.038673 12239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:38:37.039464 12237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:38:37.040308 12231 server_base.cc:1034] running on GCE node
I20240506 11:38:37.040870 12231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:38:37.041702 12231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:38:37.042901 12231 hybrid_clock.cc:648] HybridClock initialized: now 1714995517042848 us; error 37 us; skew 500 ppm
I20240506 11:38:37.043208 12231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:38:37.045563 12231 webserver.cc:460] Webserver started at http://127.10.45.131:37835/ using document root <none> and password file <none>
I20240506 11:38:37.046283 12231 fs_manager.cc:359] Metadata directory not provided
I20240506 11:38:37.046424 12231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:38:37.046726 12231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:38:37.049322 12231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e36d0b9df4204b1d8a524fd2a2f78042"
format_stamp: "Formatted at 2024-05-06 11:38:37 on dist-test-slave-7l0g"
I20240506 11:38:37.050047 12231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e36d0b9df4204b1d8a524fd2a2f78042"
format_stamp: "Formatted at 2024-05-06 11:38:37 on dist-test-slave-7l0g"
I20240506 11:38:37.055060 12231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240506 11:38:37.058328 12245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:37.059659 12231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240506 11:38:37.059816 12231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e36d0b9df4204b1d8a524fd2a2f78042"
format_stamp: "Formatted at 2024-05-06 11:38:37 on dist-test-slave-7l0g"
I20240506 11:38:37.059983 12231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:38:37.075718 12231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:38:37.076503 12231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:38:37.076767 12231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:38:37.077728 12231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:38:37.079226 12231 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:38:37.079306 12231 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:37.079437 12231 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:38:37.079490 12231 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:38:37.110324 12231 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:34507
I20240506 11:38:37.110342 12357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:34507 every 8 connection(s)
I20240506 11:38:37.111590 12231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:38:37.120491 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 12231
I20240506 11:38:37.120730 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:38:37.123708 12358 heartbeater.cc:347] Connected to a master server at 127.10.45.190:37311
I20240506 11:38:37.123991 12358 heartbeater.cc:464] Registering TS with master...
I20240506 11:38:37.124768 12358 heartbeater.cc:511] Master 127.10.45.190:37311 requested a full tablet report, sending...
I20240506 11:38:37.126004 11911 ts_manager.cc:194] Registered new tserver with Master: e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507)
I20240506 11:38:37.126730 11911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:39733
I20240506 11:38:37.136058 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:38:37.139092 10422 test_util.cc:268] Using random seed: -2104022928
I20240506 11:38:37.155313 11911 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45532:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 11:38:37.156700 11911 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 11:38:37.181300 12293 tablet_service.cc:1467] Processing CreateTablet for tablet 63d657e07d5047b98c616f9a175d5cfe (DEFAULT_TABLE table=test-workload [id=c9222443df4040368f422f8d48ebae1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:38:37.182489 12293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d657e07d5047b98c616f9a175d5cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:38:37.182528 12032 tablet_service.cc:1467] Processing CreateTablet for tablet 63d657e07d5047b98c616f9a175d5cfe (DEFAULT_TABLE table=test-workload [id=c9222443df4040368f422f8d48ebae1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:38:37.183689 12032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d657e07d5047b98c616f9a175d5cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:38:37.185367 12163 tablet_service.cc:1467] Processing CreateTablet for tablet 63d657e07d5047b98c616f9a175d5cfe (DEFAULT_TABLE table=test-workload [id=c9222443df4040368f422f8d48ebae1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:38:37.186630 12163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d657e07d5047b98c616f9a175d5cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:38:37.192816 12376 tablet_bootstrap.cc:492] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042: Bootstrap starting.
I20240506 11:38:37.195408 12376 tablet_bootstrap.cc:654] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042: Neither blocks nor log segments found. Creating new log.
I20240506 11:38:37.196079 12377 tablet_bootstrap.cc:492] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e: Bootstrap starting.
I20240506 11:38:37.196604 12376 log.cc:824] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042: Log is configured to *not* fsync() on all Append() calls
I20240506 11:38:37.198598 12379 tablet_bootstrap.cc:492] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391: Bootstrap starting.
I20240506 11:38:37.199237 12377 tablet_bootstrap.cc:654] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e: Neither blocks nor log segments found. Creating new log.
I20240506 11:38:37.200271 12376 tablet_bootstrap.cc:492] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042: No bootstrap required, opened a new log
I20240506 11:38:37.200448 12377 log.cc:824] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e: Log is configured to *not* fsync() on all Append() calls
I20240506 11:38:37.200526 12376 ts_tablet_manager.cc:1367] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240506 11:38:37.201695 12379 tablet_bootstrap.cc:654] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391: Neither blocks nor log segments found. Creating new log.
I20240506 11:38:37.202589 12379 log.cc:824] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391: Log is configured to *not* fsync() on all Append() calls
I20240506 11:38:37.203109 12377 tablet_bootstrap.cc:492] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e: No bootstrap required, opened a new log
I20240506 11:38:37.203444 12377 ts_tablet_manager.cc:1367] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240506 11:38:37.204806 12379 tablet_bootstrap.cc:492] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391: No bootstrap required, opened a new log
I20240506 11:38:37.205168 12379 ts_tablet_manager.cc:1367] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240506 11:38:37.208868 12376 raft_consensus.cc:348] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.209270 12376 raft_consensus.cc:374] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:38:37.209414 12376 raft_consensus.cc:729] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36d0b9df4204b1d8a524fd2a2f78042, State: Initialized, Role: FOLLOWER
I20240506 11:38:37.210291 12376 consensus_queue.cc:260] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.210788 12382 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:38:37.211076 12382 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.212069 12379 raft_consensus.cc:348] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.212500 12379 raft_consensus.cc:374] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:38:37.212324 12377 raft_consensus.cc:348] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.212671 12379 raft_consensus.cc:729] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1dfb32eb83944708aa5de1a681ed391, State: Initialized, Role: FOLLOWER
I20240506 11:38:37.212898 12377 raft_consensus.cc:374] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:38:37.213073 12377 raft_consensus.cc:729] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a0d74bcf824aebbbb86cf0fd12f16e, State: Initialized, Role: FOLLOWER
I20240506 11:38:37.213371 12379 consensus_queue.cc:260] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.213910 12377 consensus_queue.cc:260] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.221549 12228 heartbeater.cc:503] Master 127.10.45.190:37311 was elected leader, sending a full tablet report...
I20240506 11:38:37.221613 12379 ts_tablet_manager.cc:1398] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391: Time spent starting tablet: real 0.016s user 0.003s sys 0.005s
W20240506 11:38:37.222072 12246 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.45.130:45689 timed out after 0.008s (SENT)
I20240506 11:38:37.225777 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:37.226415 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 0.
I20240506 11:38:37.229422 12387 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:38:37.229557 12097 heartbeater.cc:503] Master 127.10.45.190:37311 was elected leader, sending a full tablet report...
I20240506 11:38:37.230520 12387 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
W20240506 11:38:37.234426 12246 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689): Timed out: RequestConsensusVote RPC to 127.10.45.130:45689 timed out after 0.008s (SENT)
I20240506 11:38:37.234910 12382 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.235662 12358 heartbeater.cc:503] Master 127.10.45.190:37311 was elected leader, sending a full tablet report...
I20240506 11:38:37.236469 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:37.236740 12182 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 0.
I20240506 11:38:37.242975 12377 ts_tablet_manager.cc:1398] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e: Time spent starting tablet: real 0.039s user 0.006s sys 0.008s
I20240506 11:38:37.244776 12387 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507)
I20240506 11:38:37.245272 12376 ts_tablet_manager.cc:1398] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042: Time spent starting tablet: real 0.045s user 0.013s sys 0.015s
I20240506 11:38:37.247303 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:37.247877 12052 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 0.
W20240506 11:38:37.248677 11985 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.45.130:45689 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20240506 11:38:37.249457 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042; no voters: f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:37.251737 12383 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 11:38:37.252207 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:38:37.252558 12383 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:38:37.256482 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.258610 12385 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240506 11:38:37.258661 12182 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.131:60989 (request call id 1) took 11 ms (client timeout 8 ms). Trace:
I20240506 11:38:37.259950 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 1 election: Requested vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.258894 12385 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.262130 12385 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
W20240506 11:38:37.262428 12182 rpcz_store.cc:269] 0506 11:38:37.225837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:37.225903 (+ 66us) service_pool.cc:225] Handling call
0506 11:38:37.236873 (+ 10970us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 11:38:37.265026 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e"
I20240506 11:38:37.265314 12052 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 0 FOLLOWER]: Advancing to term 1
W20240506 11:38:37.267719 12098 tablet.cc:2369] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 11:38:37.271318 12183 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.131:60989 (request call id 2) took 10 ms (client timeout 8 ms). Trace:
W20240506 11:38:37.271490 12183 rpcz_store.cc:269] 0506 11:38:37.260606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:37.270491 (+ 9885us) service_pool.cc:212] Skipping call since client already timed out
0506 11:38:37.271250 (+ 759us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 11:38:37.271443 12052 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 1.
W20240506 11:38:37.272115 12052 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.131:56909 (request call id 2) took 7 ms (client timeout 8 ms). Trace:
I20240506 11:38:37.272419 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391"
I20240506 11:38:37.272619 12182 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 0 FOLLOWER]: Advancing to term 1
W20240506 11:38:37.272338 12052 rpcz_store.cc:269] 0506 11:38:37.264788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:37.264850 (+ 62us) service_pool.cc:225] Handling call
0506 11:38:37.272071 (+ 7221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 11:38:37.273815 12181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:37.275652 12182 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 1.
I20240506 11:38:37.276396 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:38:37.276861 12383 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:38:37.277151 12383 raft_consensus.cc:686] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 1 LEADER]: Becoming Leader. State: Replica: e36d0b9df4204b1d8a524fd2a2f78042, State: Running, Role: LEADER
I20240506 11:38:37.277832 12383 consensus_queue.cc:237] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
W20240506 11:38:37.278693 11986 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507): Timed out: connection negotiation to 127.10.45.131:34507 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20240506 11:38:37.278893 11986 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6a0d74bcf824aebbbb86cf0fd12f16e; no voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:37.279379 12387 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 11:38:37.286298 12387 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:38:37.286473 12387 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.287238 12387 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507)
W20240506 11:38:37.288973 12247 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.008s (SENT)
I20240506 11:38:37.299593 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:37.299871 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a0d74bcf824aebbbb86cf0fd12f16e in term 1.
I20240506 11:38:37.300478 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e"
I20240506 11:38:37.300843 12052 raft_consensus.cc:2362] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 1. Re-sending same reply.
I20240506 11:38:37.301353 11985 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:38:37.301759 12387 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 11:38:37.301877 12387 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:38:37.301967 12387 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 1 FOLLOWER]: Advancing to term 2
W20240506 11:38:37.303406 11986 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507): Timed out: RequestConsensusVote RPC to 127.10.45.131:34507 timed out after 0.007s (SENT)
I20240506 11:38:37.305680 12387 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.307646 12387 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 2 election: Requested vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507)
I20240506 11:38:37.308452 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391"
I20240506 11:38:37.309723 12052 raft_consensus.cc:1226] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 2 FOLLOWER]: Rejecting Update request from peer e36d0b9df4204b1d8a524fd2a2f78042 for earlier term 1. Current term is 2. Ops: []
I20240506 11:38:37.312955 11911 catalog_manager.cc:5484] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 reported cstate change: term changed from 0 to 1, leader changed from <none> to e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131). New cstate: current_term: 1 leader_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } } }
I20240506 11:38:37.314353 12382 consensus_queue.cc:1046] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:38:37.315202 12395 raft_consensus.cc:3032] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:38:37.315316 12395 raft_consensus.cc:729] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e36d0b9df4204b1d8a524fd2a2f78042, State: Running, Role: LEADER
I20240506 11:38:37.315646 12395 consensus_queue.cc:260] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
W20240506 11:38:37.316553 12117 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.45.131:34507 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20240506 11:38:37.317176 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:37.317204 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:37.318203 12311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042"
I20240506 11:38:37.318472 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042"
I20240506 11:38:37.318984 11986 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6a0d74bcf824aebbbb86cf0fd12f16e; no voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:37.319295 12395 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:38:37.319461 12387 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 11:38:37.320190 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:37.320286 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:37.320519 12052 raft_consensus.cc:2354] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 for earlier term 1. Current term is 2.
I20240506 11:38:37.321084 12117 leader_election.cc:398] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 1 pre-election: Vote denied by peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317) with higher term. Message: Invalid argument: T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 for earlier term 1. Current term is 2.
I20240506 11:38:37.321167 12117 leader_election.cc:401] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20240506 11:38:37.321434 12385 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:38:37.322036 12382 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:38:37.322161 12382 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.322507 12312 raft_consensus.cc:2396] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6a0d74bcf824aebbbb86cf0fd12f16e for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240506 11:38:37.322811 12382 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
W20240506 11:38:37.323127 12312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.129:51629 (request call id 3) took 6 ms (client timeout 7 ms). Trace:
I20240506 11:38:37.323253 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:37.323876 12385 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317) with higher term. Message: Invalid argument: T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 for earlier term 1. Current term is 2.
I20240506 11:38:37.324121 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 2.
W20240506 11:38:37.323251 12312 rpcz_store.cc:269] 0506 11:38:37.316897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:37.316940 (+ 43us) service_pool.cc:225] Handling call
0506 11:38:37.322987 (+ 6047us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x616000071554. stack: 00007fcdabed5756 00007fcdabed5510 00007fcdaaf0848d 0000000000955804 00000000009557cd 000000000095519a 00007fcdb77346b5 00007fcdbb406b04 00007fcdb5b0a9b6 00007fcdb5b0a6e5 00007fcdb2bdd04e 00007fcdb2bdbf2e 00007fcdb2be2405 00007fcdb2be2d2f 00007fcdb2be2b58 0000000000a60bcc
0506 11:38:37.323063 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11213568}
I20240506 11:38:37.324460 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:38:37.324710 12382 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 11:38:37.324798 12382 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:38:37.324883 12382 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 2 FOLLOWER]: Advancing to term 3
I20240506 11:38:37.325122 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:37.325305 12052 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 2.
I20240506 11:38:37.327579 12382 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.329164 12382 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 3 election: Requested vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.329618 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e"
I20240506 11:38:37.329770 12052 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 2 FOLLOWER]: Advancing to term 3
I20240506 11:38:37.330621 12385 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:38:37.330852 12385 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.331164 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391"
I20240506 11:38:37.331342 12183 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 2 FOLLOWER]: Advancing to term 3
I20240506 11:38:37.331930 12052 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 3.
I20240506 11:38:37.332352 12385 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.332446 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042; no voters:
I20240506 11:38:37.332928 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:37.333199 12312 raft_consensus.cc:2379] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 in current term 3: Already voted for candidate e36d0b9df4204b1d8a524fd2a2f78042 in this term.
I20240506 11:38:37.333612 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:37.333855 12052 raft_consensus.cc:2379] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 in current term 3: Already voted for candidate e36d0b9df4204b1d8a524fd2a2f78042 in this term.
I20240506 11:38:37.334086 12382 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 11:38:37.334236 12382 raft_consensus.cc:686] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 3 LEADER]: Becoming Leader. State: Replica: e36d0b9df4204b1d8a524fd2a2f78042, State: Running, Role: LEADER
I20240506 11:38:37.334280 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 3.
I20240506 11:38:37.334277 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
W20240506 11:38:37.334431 12183 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.131:60989 (request call id 7) took 5 ms (client timeout 6 ms). Trace:
W20240506 11:38:37.334532 12183 rpcz_store.cc:269] 0506 11:38:37.329290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:37.331024 (+ 1734us) service_pool.cc:225] Handling call
0506 11:38:37.334385 (+ 3361us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 11:38:37.334496 12382 consensus_queue.cc:237] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.334996 12385 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240506 11:38:37.336886 12183 raft_consensus.cc:1261] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 3 FOLLOWER]: Refusing update from remote peer e36d0b9df4204b1d8a524fd2a2f78042: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 11:38:37.337349 12395 consensus_queue.cc:1035] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:38:37.337821 12052 raft_consensus.cc:1261] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 3 FOLLOWER]: Refusing update from remote peer e36d0b9df4204b1d8a524fd2a2f78042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240506 11:38:37.337849 11910 catalog_manager.cc:5484] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } health_report { overall_health: UNKNOWN } } }
I20240506 11:38:37.338315 12382 consensus_queue.cc:1035] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:38:37.359369 12385 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e36d0b9df4204b1d8a524fd2a2f78042)
I20240506 11:38:37.359557 12385 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.360481 12385 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.361726 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:37.362016 12051 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1dfb32eb83944708aa5de1a681ed391 in term 3.
I20240506 11:38:37.362422 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:38:37.362515 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:37.362684 12384 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 11:38:37.362787 12384 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e36d0b9df4204b1d8a524fd2a2f78042)
I20240506 11:38:37.362898 12384 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 3 FOLLOWER]: Advancing to term 4
I20240506 11:38:37.363227 12387 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e36d0b9df4204b1d8a524fd2a2f78042)
I20240506 11:38:37.363401 12387 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.364307 12387 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507)
I20240506 11:38:37.364678 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:37.365795 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:37.365864 12384 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.367146 12183 raft_consensus.cc:2379] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6a0d74bcf824aebbbb86cf0fd12f16e in current term 4: Already voted for candidate f1dfb32eb83944708aa5de1a681ed391 in this term.
I20240506 11:38:37.367808 12182 raft_consensus.cc:1226] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 4 FOLLOWER]: Rejecting Update request from peer e36d0b9df4204b1d8a524fd2a2f78042 for earlier term 3. Current term is 4. Ops: []
I20240506 11:38:37.368065 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042"
I20240506 11:38:37.368584 12382 consensus_queue.cc:1046] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:38:37.369361 11985 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6a0d74bcf824aebbbb86cf0fd12f16e; no voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:37.369626 12386 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 3 FOLLOWER]: Advancing to term 4
I20240506 11:38:37.370554 12182 raft_consensus.cc:1226] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 4 FOLLOWER]: Rejecting Update request from peer e36d0b9df4204b1d8a524fd2a2f78042 for earlier term 3. Current term is 4. Ops: []
I20240506 11:38:37.371311 12395 consensus_queue.cc:1046] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:38:37.372596 12386 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 11:38:37.372821 12399 raft_consensus.cc:3032] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 3 LEADER]: Stepping down as leader of term 3
I20240506 11:38:37.372905 12399 raft_consensus.cc:729] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e36d0b9df4204b1d8a524fd2a2f78042, State: Running, Role: LEADER
I20240506 11:38:37.372910 12182 raft_consensus.cc:1226] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 4 FOLLOWER]: Rejecting Update request from peer e36d0b9df4204b1d8a524fd2a2f78042 for earlier term 3. Current term is 4. Ops: []
I20240506 11:38:37.373260 12399 consensus_queue.cc:260] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
W20240506 11:38:37.373540 12272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47926: Illegal state: replica e36d0b9df4204b1d8a524fd2a2f78042 is not leader of this config: current role FOLLOWER
I20240506 11:38:37.373582 12382 consensus_queue.cc:1046] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:38:37.375334 12399 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 3 FOLLOWER]: Advancing to term 4
I20240506 11:38:37.377143 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042"
W20240506 11:38:37.379287 12406 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47926: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.007s ago.
I20240506 11:38:37.379848 12384 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 4 election: Requested vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.382045 12410 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:38:37.382179 12410 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.382896 12410 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507)
I20240506 11:38:37.383205 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
W20240506 11:38:37.383297 12117 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507): Timed out: RequestConsensusVote RPC to 127.10.45.131:34507 timed out after 0.006s (SENT)
I20240506 11:38:37.383322 12312 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1dfb32eb83944708aa5de1a681ed391 in term 4.
I20240506 11:38:37.383422 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a0d74bcf824aebbbb86cf0fd12f16e in term 4.
W20240506 11:38:37.383590 12312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.130:47833 (request call id 5) took 6 ms (client timeout 6 ms). Trace:
I20240506 11:38:37.383709 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:38:37.383749 11985 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:38:37.383859 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.383994 12410 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 11:38:37.384135 12410 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:38:37.384248 12410 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 4 FOLLOWER]: Advancing to term 5
I20240506 11:38:37.384227 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e"
I20240506 11:38:37.384703 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.385286 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
W20240506 11:38:37.383730 12312 rpcz_store.cc:269] 0506 11:38:37.376950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:37.377032 (+ 82us) service_pool.cc:225] Handling call
0506 11:38:37.383523 (+ 6491us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x616000071554. stack: 00007fcdabed5756 00007fcdabed5510 00007fcdaaf0848d 0000000000955804 00000000009557cd 000000000095519a 00007fcdb77346b5 00007fcdbb406b04 00007fcdb5b0a9b6 00007fcdb5b0a6e5 00007fcdb2bdd04e 00007fcdb2bdbf2e 00007fcdb2be2405 00007fcdb2be2d2f 00007fcdb2be2b58 0000000000a60bcc
0506 11:38:37.383557 (+ 34us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4551424}
I20240506 11:38:37.386435 12410 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.386454 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:37.386675 12313 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a0d74bcf824aebbbb86cf0fd12f16e in term 4.
I20240506 11:38:37.387058 12410 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 5 election: Requested vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507)
I20240506 11:38:37.387226 12051 raft_consensus.cc:2354] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 for earlier term 4. Current term is 5.
I20240506 11:38:37.387329 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:37.387547 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 4.
I20240506 11:38:37.387566 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042"
I20240506 11:38:37.387748 12313 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 4 FOLLOWER]: Advancing to term 5
I20240506 11:38:37.387951 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e
W20240506 11:38:37.387982 12051 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.130:36663 (request call id 4) took 7 ms (client timeout 6 ms). Trace:
W20240506 11:38:37.388231 12051 rpcz_store.cc:269] 0506 11:38:37.380329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:37.384044 (+ 3715us) service_pool.cc:225] Handling call
0506 11:38:37.387799 (+ 3755us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x616000071554. stack: 00007f4bf0d1f756 00007f4bf0d1f510 00007f4befd5248d 0000000000955804 00000000009557cd 000000000095519a 00007f4bfc57e6b5 00007f4c00250b04 00007f4bfa9549b6 00007f4bfa9546e5 00007f4bf7a2704e 00007f4bf7a25f2e 00007f4bf7a2c405 00007f4bf7a2cd2f 00007f4bf7a2cb58 0000000000a60bcc
0506 11:38:37.387922 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6235520}
I20240506 11:38:37.389828 12313 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a0d74bcf824aebbbb86cf0fd12f16e in term 5.
I20240506 11:38:37.389899 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e"
I20240506 11:38:37.390190 12052 raft_consensus.cc:2354] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 for earlier term 4. Current term is 5.
I20240506 11:38:37.390172 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391"
I20240506 11:38:37.390256 11986 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042; no voters:
I20240506 11:38:37.390492 12383 raft_consensus.cc:2741] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240506 11:38:37.390689 12117 leader_election.cc:398] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 4 election: Vote denied by peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317) with higher term. Message: Invalid argument: T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 for earlier term 4. Current term is 5.
I20240506 11:38:37.390782 12117 leader_election.cc:401] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240506 11:38:37.390455 12183 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 4 FOLLOWER]: Advancing to term 5
W20240506 11:38:37.392844 12143 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41786: Illegal state: replica f1dfb32eb83944708aa5de1a681ed391 is not leader of this config: current role FOLLOWER
I20240506 11:38:37.393013 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a0d74bcf824aebbbb86cf0fd12f16e in term 5.
I20240506 11:38:37.393060 12410 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 5 FOLLOWER]: Leader election won for term 5
I20240506 11:38:37.393213 12384 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317) with higher term. Message: Invalid argument: T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 for earlier term 4. Current term is 5.
I20240506 11:38:37.393316 12410 raft_consensus.cc:686] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 5 LEADER]: Becoming Leader. State: Replica: b6a0d74bcf824aebbbb86cf0fd12f16e, State: Running, Role: LEADER
W20240506 11:38:37.392808 12142 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41786: Illegal state: replica f1dfb32eb83944708aa5de1a681ed391 is not leader of this config: current role FOLLOWER
I20240506 11:38:37.393759 12410 consensus_queue.cc:237] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.396560 12183 raft_consensus.cc:1261] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Refusing update from remote peer b6a0d74bcf824aebbbb86cf0fd12f16e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 11:38:37.396943 12313 raft_consensus.cc:1261] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Refusing update from remote peer b6a0d74bcf824aebbbb86cf0fd12f16e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240506 11:38:37.396670 11911 catalog_manager.cc:5484] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e reported cstate change: term changed from 3 to 5, leader changed from e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131) to b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129). New cstate: current_term: 5 leader_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } health_report { overall_health: HEALTHY } } }
I20240506 11:38:37.397111 12386 consensus_queue.cc:1035] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:38:37.397466 12410 consensus_queue.cc:1035] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:38:37.398355 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:38:37.398494 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.400270 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.401208 12384 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:38:37.401706 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:37.401350 12384 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.402200 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:37.402916 12384 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.403455 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:37.403789 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:37.404062 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:37.404525 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:37.405426 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:37.405689 12384 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 11:38:37.412315 12409 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240506 11:38:37.412396 12409 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240506 11:38:37.518674 12384 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:37.518875 12384 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.519726 12384 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.520265 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:37.519994 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:37.520876 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:37.521131 12384 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 11:38:37.882501 12312 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240506 11:38:37.882531 12182 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240506 11:38:37.882632 12182 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20240506 11:38:37.882633 12312 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240506 11:38:37.948963 12384 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:37.949188 12384 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:37.950013 12384 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:37.950830 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:37.950896 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:37.951682 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:37.952288 12384 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:38.640270 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:38.640480 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:38.641237 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:38.641716 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:38.641722 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:38.642318 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:39.082811 12384 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:39.083012 12384 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:39.083695 12384 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:39.084235 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:39.084312 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:39.084996 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:39.085304 12384 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:39.086982 12421 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=0, attempt_no=2}) took 1680 ms. Trace:
I20240506 11:38:39.087158 12421 rpcz_store.cc:276] 0506 11:38:37.406850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:37.410246 (+ 3396us) service_pool.cc:225] Handling call
0506 11:38:39.086906 (+1676660us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:37.881386 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:37.881499 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:37.881505 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:37.881507 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:37.881596 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:37.881633 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:37.881637 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:37.881711 (+ 74us) tablet.cc:699] Row locks acquired
0506 11:38:37.881714 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:38:37.881753 (+ 39us) write_op.cc:270] Start()
0506 11:38:38.203904 (+322151us) write_op.cc:276] Timestamp: P: 1714995517881715 usec, L: 0
0506 11:38:38.203908 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:38:38.204145 (+ 237us) log.cc:842] Serialized 345 byte log entry
0506 11:38:39.084200 (+880055us) op_driver.cc:444] REPLICATION: finished
0506 11:38:39.085165 (+ 965us) write_op.cc:301] APPLY: starting
0506 11:38:39.085186 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:39.085229 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:39.085254 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:39.085338 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:39.085377 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:39.085384 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:39.085469 (+ 85us) log.cc:842] Serialized 22 byte log entry
0506 11:38:39.086773 (+ 1304us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:39.086797 (+ 24us) write_op.cc:454] Released schema lock
0506 11:38:39.086805 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":896,"num_ops":1,"prepare.queue_time_us":470624,"prepare.run_cpu_time_us":743,"prepare.run_wall_time_us":322833,"replication_time_us":880263,"thread_start_us":820,"threads_started":1}]]}
W20240506 11:38:39.088264 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.977s user 0.000s sys 0.000s
W20240506 11:38:39.088285 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.978s user 0.000s sys 0.001s
I20240506 11:38:39.762586 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:39.763041 12312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:51629 (request call id 353) took 1121 ms. Trace:
I20240506 11:38:39.763166 12312 rpcz_store.cc:276] 0506 11:38:38.641825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:38.642085 (+ 260us) service_pool.cc:225] Handling call
0506 11:38:38.642328 (+ 243us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 11:38:38.642422 (+ 94us) raft_consensus.cc:1520] Early marking committed up to index 4
0506 11:38:38.642742 (+ 320us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 11:38:38.642871 (+ 129us) write_op.cc:270] Start()
0506 11:38:39.044046 (+401175us) write_op.cc:276] Timestamp: P: 1714995517881715 usec, L: 0
0506 11:38:39.044405 (+ 359us) write_op.cc:270] Start()
0506 11:38:39.761600 (+717195us) write_op.cc:276] Timestamp: P: 1714995518204535 usec, L: 0
0506 11:38:39.762159 (+ 559us) log.cc:842] Serialized 699 byte log entry
0506 11:38:39.762464 (+ 305us) raft_consensus.cc:1631] Marking committed up to 4
0506 11:38:39.762473 (+ 9us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:38:39.762510 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:38:39.762918 (+ 408us) raft_consensus.cc:1671] finished
0506 11:38:39.762923 (+ 5us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:38:39.763008 (+ 85us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:39.044289 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:39.044395 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:39.044400 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:38:39.044402 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:39.044476 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:39.044489 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:39.044493 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:39.044537 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:38:39.044539 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0506 11:38:39.762052 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:39.762154 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:39.762160 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:39.762162 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:39.762240 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:39.762272 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:39.762275 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:39.762324 (+ 49us) tablet.cc:699] Row locks acquired
0506 11:38:39.762325 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":224,"threads_started":1,"wal-append.queue_time_us":301,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":82,"prepare.run_cpu_time_us":329,"prepare.run_wall_time_us":330}],["op",{"num_ops":1,"partition_lock_wait_us":25,"prepare.queue_time_us":308,"prepare.run_cpu_time_us":335,"prepare.run_wall_time_us":336,"thread_start_us":195,"threads_started":1}]]}
I20240506 11:38:39.765712 12428 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=2, attempt_no=0}) took 1808 ms. Trace:
I20240506 11:38:39.765918 12428 rpcz_store.cc:276] 0506 11:38:37.957094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:37.957295 (+ 201us) service_pool.cc:225] Handling call
0506 11:38:39.765668 (+1808373us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:38.204251 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:38.204370 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:38.204376 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:38.204378 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:38.204475 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:38.204487 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:38.204491 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:38.204559 (+ 68us) tablet.cc:699] Row locks acquired
0506 11:38:38.204562 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:38:38.204607 (+ 45us) write_op.cc:270] Start()
0506 11:38:38.552808 (+348201us) write_op.cc:276] Timestamp: P: 1714995518204535 usec, L: 0
0506 11:38:38.552817 (+ 9us) op_driver.cc:331] REPLICATION: starting
0506 11:38:38.553108 (+ 291us) log.cc:842] Serialized 354 byte log entry
0506 11:38:39.764715 (+1211607us) op_driver.cc:444] REPLICATION: finished
0506 11:38:39.765159 (+ 444us) write_op.cc:301] APPLY: starting
0506 11:38:39.765176 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:39.765209 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:39.765239 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:39.765329 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:39.765354 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:39.765359 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:38:39.765435 (+ 76us) log.cc:842] Serialized 22 byte log entry
0506 11:38:39.765541 (+ 106us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:39.765559 (+ 18us) write_op.cc:454] Released schema lock
0506 11:38:39.765566 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":381,"num_ops":1,"prepare.queue_time_us":246416,"prepare.run_cpu_time_us":847,"prepare.run_wall_time_us":348948,"replication_time_us":1211875,"thread_start_us":317,"threads_started":1}]]}
I20240506 11:38:39.861979 12384 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:39.862181 12384 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:39.863001 12384 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:39.863415 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:39.863895 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:39.864502 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:40.409406 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:40.409636 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:40.410351 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:40.410815 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:40.410979 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:40.411535 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:40.412313 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:40.414234 12011 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 8) took 1174 ms. Trace:
I20240506 11:38:40.414507 12437 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=3, attempt_no=0}) took 1325 ms. Trace:
I20240506 11:38:40.414352 12011 rpcz_store.cc:276] 0506 11:38:39.239939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:39.240016 (+ 77us) service_pool.cc:225] Handling call
0506 11:38:39.240292 (+ 276us) tablet_service.cc:2780] Created scanner b990feb911474dc28e07e7dbf776d428 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is e2573c1258f043a080d0fed565f92ab7
0506 11:38:39.241281 (+ 989us) tablet_service.cc:2920] Creating iterator
0506 11:38:39.241325 (+ 44us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:39.241337 (+ 12us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:40.412537 (+1171200us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1171195 microseconds
0506 11:38:40.412641 (+ 104us) tablet_service.cc:2945] Iterator created
0506 11:38:40.413134 (+ 493us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:40.413179 (+ 45us) tablet_service.cc:3010] has_more: true
0506 11:38:40.413219 (+ 40us) tablet_service.cc:3027] Continuing scan request
0506 11:38:40.413338 (+ 119us) tablet_service.cc:3091] Found scanner b990feb911474dc28e07e7dbf776d428 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is e2573c1258f043a080d0fed565f92ab7
0506 11:38:40.414158 (+ 820us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":45184}
I20240506 11:38:40.414634 12437 rpcz_store.cc:276] 0506 11:38:39.089473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:39.089553 (+ 80us) service_pool.cc:225] Handling call
0506 11:38:40.414448 (+1324895us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:39.090381 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:39.090455 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:39.090459 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:38:39.090460 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:39.090531 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:39.090539 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:39.090542 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:39.090584 (+ 42us) tablet.cc:699] Row locks acquired
0506 11:38:39.090585 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:39.090614 (+ 29us) write_op.cc:270] Start()
0506 11:38:39.187756 (+ 97142us) write_op.cc:276] Timestamp: P: 1714995519090568 usec, L: 0
0506 11:38:39.187761 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:38:39.188011 (+ 250us) log.cc:842] Serialized 354 byte log entry
0506 11:38:40.411113 (+1223102us) op_driver.cc:444] REPLICATION: finished
0506 11:38:40.411903 (+ 790us) write_op.cc:301] APPLY: starting
0506 11:38:40.411942 (+ 39us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:40.411981 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:40.412014 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:40.412083 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:40.412125 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:40.412135 (+ 10us) write_op.cc:312] APPLY: finished
0506 11:38:40.412242 (+ 107us) log.cc:842] Serialized 22 byte log entry
0506 11:38:40.414325 (+ 2083us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:40.414351 (+ 26us) write_op.cc:454] Released schema lock
0506 11:38:40.414358 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":689,"num_ops":1,"prepare.queue_time_us":372,"prepare.run_cpu_time_us":646,"prepare.run_wall_time_us":97710,"replication_time_us":1223328,"thread_start_us":842,"threads_started":2}]]}
W20240506 11:38:40.415201 12404 scanner-internal.cc:458] Time spent opening tablet: real 1.176s user 0.001s sys 0.000s
I20240506 11:38:40.416019 12012 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 9) took 1175 ms. Trace:
I20240506 11:38:40.416116 12012 rpcz_store.cc:276] 0506 11:38:39.240042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:39.240112 (+ 70us) service_pool.cc:225] Handling call
0506 11:38:39.240292 (+ 180us) tablet_service.cc:2780] Created scanner 1203e84a0f2440e8b3620edf7fd96428 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 2fd1833718094c768fa4efb582bfa3dd
0506 11:38:39.241281 (+ 989us) tablet_service.cc:2920] Creating iterator
0506 11:38:39.241325 (+ 44us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:39.241342 (+ 17us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:40.414362 (+1173020us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1173023 microseconds
0506 11:38:40.414455 (+ 93us) tablet_service.cc:2945] Iterator created
0506 11:38:40.415002 (+ 547us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:40.415035 (+ 33us) tablet_service.cc:3010] has_more: true
0506 11:38:40.415068 (+ 33us) tablet_service.cc:3027] Continuing scan request
0506 11:38:40.415140 (+ 72us) tablet_service.cc:3091] Found scanner 1203e84a0f2440e8b3620edf7fd96428 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 2fd1833718094c768fa4efb582bfa3dd
0506 11:38:40.415979 (+ 839us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":86784}
W20240506 11:38:40.416613 12403 scanner-internal.cc:458] Time spent opening tablet: real 1.177s user 0.001s sys 0.000s
I20240506 11:38:40.822440 12384 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:41.221872 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:41.222069 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:41.222811 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:41.223260 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:41.223330 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:41.224189 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:41.224591 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:41.225317 12442 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=4, attempt_no=0}) took 1457 ms. Trace:
I20240506 11:38:41.225494 12442 rpcz_store.cc:276] 0506 11:38:39.767702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:39.767774 (+ 72us) service_pool.cc:225] Handling call
0506 11:38:41.225252 (+1457478us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:39.768464 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:39.768547 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:39.768551 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:38:39.768552 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:39.768615 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:39.768623 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:39.768626 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:39.768673 (+ 47us) tablet.cc:699] Row locks acquired
0506 11:38:39.768674 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:39.768705 (+ 31us) write_op.cc:270] Start()
0506 11:38:40.732831 (+964126us) write_op.cc:276] Timestamp: P: 1714995519768628 usec, L: 0
0506 11:38:40.732835 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:38:40.733148 (+ 313us) log.cc:842] Serialized 354 byte log entry
0506 11:38:41.222770 (+489622us) op_driver.cc:444] REPLICATION: finished
0506 11:38:41.223240 (+ 470us) write_op.cc:301] APPLY: starting
0506 11:38:41.223257 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:41.223297 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:41.223330 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:41.223395 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:41.223427 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:41.223434 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:41.223504 (+ 70us) log.cc:842] Serialized 22 byte log entry
0506 11:38:41.225127 (+ 1623us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:41.225145 (+ 18us) write_op.cc:454] Released schema lock
0506 11:38:41.225152 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":404,"num_ops":1,"prepare.queue_time_us":330,"prepare.run_cpu_time_us":712,"prepare.run_wall_time_us":964776,"raft.queue_time_us":37,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":489909,"spinlock_wait_cycles":1280,"thread_start_us":556,"threads_started":2}]]}
W20240506 11:38:41.226027 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.660s user 0.000s sys 0.000s
W20240506 11:38:41.226518 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.659s user 0.000s sys 0.000s
I20240506 11:38:41.502660 12384 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:41.502887 12384 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:41.503679 12384 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:41.504184 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:41.504240 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:41.504819 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:41.603566 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:41.603833 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:41.604722 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:41.605196 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:41.605351 12442 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=5, attempt_no=0}) took 1188 ms. Trace:
I20240506 11:38:41.605443 12442 rpcz_store.cc:276] 0506 11:38:40.416691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:40.416733 (+ 42us) service_pool.cc:225] Handling call
0506 11:38:41.605291 (+1188558us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:40.733278 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:40.733433 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:40.733440 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:38:40.733444 (+ 4us) tablet.cc:660] Decoding operations
0506 11:38:40.733527 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:40.733537 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:40.733540 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:40.733582 (+ 42us) tablet.cc:699] Row locks acquired
0506 11:38:40.733583 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:40.733617 (+ 34us) write_op.cc:270] Start()
0506 11:38:40.875778 (+142161us) write_op.cc:276] Timestamp: P: 1714995520733555 usec, L: 0
0506 11:38:40.875782 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:38:40.875992 (+ 210us) log.cc:842] Serialized 354 byte log entry
0506 11:38:41.604652 (+728660us) op_driver.cc:444] REPLICATION: finished
0506 11:38:41.604777 (+ 125us) write_op.cc:301] APPLY: starting
0506 11:38:41.604801 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:41.604845 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:41.604884 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:41.604954 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:41.604987 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:41.604993 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:38:41.605085 (+ 92us) log.cc:842] Serialized 22 byte log entry
0506 11:38:41.605156 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:41.605175 (+ 19us) write_op.cc:454] Released schema lock
0506 11:38:41.605201 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":68,"num_ops":1,"prepare.queue_time_us":316132,"prepare.run_cpu_time_us":707,"prepare.run_wall_time_us":142784,"replication_time_us":728841}]]}
I20240506 11:38:41.606173 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 9 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:41.606777 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:41.607053 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:42.045007 12384 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:42.132316 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:42.132504 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:42.133217 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:42.133672 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:42.133961 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:42.134552 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
W20240506 11:38:42.137368 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.761s user 0.000s sys 0.000s
W20240506 11:38:42.137377 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.760s user 0.001s sys 0.000s
I20240506 11:38:42.703796 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:42.706506 12456 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=7, attempt_no=0}) took 1099 ms. Trace:
I20240506 11:38:42.706642 12456 rpcz_store.cc:276] 0506 11:38:41.607328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:41.607375 (+ 47us) service_pool.cc:225] Handling call
0506 11:38:42.706450 (+1099075us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:41.662440 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:41.662597 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:41.662605 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:38:41.662607 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:41.662700 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:41.662713 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:41.662718 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:41.662786 (+ 68us) tablet.cc:699] Row locks acquired
0506 11:38:41.662788 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:41.662835 (+ 47us) write_op.cc:270] Start()
0506 11:38:41.676024 (+ 13189us) write_op.cc:276] Timestamp: P: 1714995521662774 usec, L: 0
0506 11:38:41.676028 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:38:41.676272 (+ 244us) log.cc:842] Serialized 354 byte log entry
0506 11:38:42.705659 (+1029387us) op_driver.cc:444] REPLICATION: finished
0506 11:38:42.705997 (+ 338us) write_op.cc:301] APPLY: starting
0506 11:38:42.706011 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:42.706046 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:42.706071 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:42.706152 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:42.706176 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:42.706181 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:38:42.706260 (+ 79us) log.cc:842] Serialized 22 byte log entry
0506 11:38:42.706336 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:42.706354 (+ 18us) write_op.cc:454] Released schema lock
0506 11:38:42.706361 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":272,"num_ops":1,"prepare.queue_time_us":54663,"prepare.run_cpu_time_us":836,"prepare.run_wall_time_us":13915,"replication_time_us":1029603,"thread_start_us":208,"threads_started":1}]]}
I20240506 11:38:42.806581 12384 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:42.806794 12384 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:42.807570 12384 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:42.808053 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 11 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:42.808236 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 11 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:42.808881 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:43.014134 12384 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 11:38:43.018095 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.730s user 0.000s sys 0.000s
W20240506 11:38:43.018148 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.730s user 0.001s sys 0.000s
I20240506 11:38:43.503904 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:43.504113 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:43.504881 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:43.505414 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:43.505625 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:43.506083 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:43.506569 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:43.697679 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:43.697886 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:43.698596 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:43.699136 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:43.699280 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:43.699774 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:43.992269 12462 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:43.992525 12462 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:43.993930 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:43.995043 12462 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:43.995128 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:43.995486 12464 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=9, attempt_no=0}) took 1286 ms. Trace:
I20240506 11:38:43.995627 12464 rpcz_store.cc:276] 0506 11:38:42.708474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:42.708531 (+ 57us) service_pool.cc:225] Handling call
0506 11:38:43.995335 (+1286804us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:42.708945 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:42.709043 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:42.709047 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:38:42.709049 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:42.709111 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:42.709119 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:42.709122 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:42.709157 (+ 35us) tablet.cc:699] Row locks acquired
0506 11:38:42.709158 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:42.709187 (+ 29us) write_op.cc:270] Start()
0506 11:38:43.696331 (+987144us) write_op.cc:276] Timestamp: P: 1714995522709138 usec, L: 0
0506 11:38:43.696338 (+ 7us) op_driver.cc:331] REPLICATION: starting
0506 11:38:43.696653 (+ 315us) log.cc:842] Serialized 354 byte log entry
0506 11:38:43.992999 (+296346us) op_driver.cc:444] REPLICATION: finished
0506 11:38:43.994132 (+ 1133us) write_op.cc:301] APPLY: starting
0506 11:38:43.994175 (+ 43us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:43.994238 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:43.994286 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:43.994419 (+ 133us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:43.994465 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:43.994481 (+ 16us) write_op.cc:312] APPLY: finished
0506 11:38:43.994749 (+ 268us) log.cc:842] Serialized 22 byte log entry
0506 11:38:43.995192 (+ 443us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:43.995212 (+ 20us) write_op.cc:454] Released schema lock
0506 11:38:43.995219 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1045,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":987782,"replication_time_us":296631,"thread_start_us":919,"threads_started":1}]]}
W20240506 11:38:43.997555 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.829s user 0.000s sys 0.000s
W20240506 11:38:43.997686 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.829s user 0.001s sys 0.000s
I20240506 11:38:43.998138 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:44.121253 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:44.666638 12462 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:44.668957 12471 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=10, attempt_no=0}) took 1651 ms. Trace:
I20240506 11:38:44.669066 12471 rpcz_store.cc:276] 0506 11:38:43.017037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:43.017099 (+ 62us) service_pool.cc:225] Handling call
0506 11:38:44.668901 (+1651802us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:43.696768 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:43.696913 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:43.696920 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:38:43.696923 (+ 3us) tablet.cc:660] Decoding operations
0506 11:38:43.697012 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:43.697024 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:43.697029 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:43.697079 (+ 50us) tablet.cc:699] Row locks acquired
0506 11:38:43.697082 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:38:43.697118 (+ 36us) write_op.cc:270] Start()
0506 11:38:43.938307 (+241189us) write_op.cc:276] Timestamp: P: 1714995523697061 usec, L: 0
0506 11:38:43.938312 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:38:43.938535 (+ 223us) log.cc:842] Serialized 354 byte log entry
0506 11:38:44.667964 (+729429us) op_driver.cc:444] REPLICATION: finished
0506 11:38:44.668353 (+ 389us) write_op.cc:301] APPLY: starting
0506 11:38:44.668372 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:44.668413 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:44.668445 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:44.668593 (+ 148us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:44.668626 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:44.668631 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:38:44.668704 (+ 73us) log.cc:842] Serialized 22 byte log entry
0506 11:38:44.668787 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:44.668805 (+ 18us) write_op.cc:454] Released schema lock
0506 11:38:44.668812 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":305,"num_ops":1,"prepare.queue_time_us":679303,"prepare.run_cpu_time_us":772,"prepare.run_wall_time_us":241838,"replication_time_us":729628,"thread_start_us":207,"threads_started":1}]]}
I20240506 11:38:44.721004 12462 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:44.721266 12462 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:44.721937 12462 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:44.722438 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 15 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:44.722501 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 15 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:44.723321 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:44.723731 12462 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 11:38:44.724877 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.576s user 0.000s sys 0.000s
W20240506 11:38:44.724897 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.577s user 0.001s sys 0.000s
I20240506 11:38:45.093616 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:45.093799 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:45.094642 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:45.095171 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 14 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:45.094971 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 14 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:45.096074 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:45.100778 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:46.434909 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:46.435133 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:46.435806 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:46.436219 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:46.436407 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:46.436945 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:46.437213 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:46.437844 12480 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=12, attempt_no=0}) took 1767 ms. Trace:
I20240506 11:38:46.438011 12480 rpcz_store.cc:276] 0506 11:38:44.670685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:44.670752 (+ 67us) service_pool.cc:225] Handling call
0506 11:38:46.437789 (+1767037us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:44.671217 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:44.671325 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:44.671328 (+ 3us) write_op.cc:435] Acquired schema lock
0506 11:38:44.671330 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:44.671393 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:44.671401 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:44.671404 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:44.671446 (+ 42us) tablet.cc:699] Row locks acquired
0506 11:38:44.671447 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:44.671478 (+ 31us) write_op.cc:270] Start()
0506 11:38:45.556600 (+885122us) write_op.cc:276] Timestamp: P: 1714995524671439 usec, L: 0
0506 11:38:45.556604 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:38:45.556825 (+ 221us) log.cc:842] Serialized 354 byte log entry
0506 11:38:46.436507 (+879682us) op_driver.cc:444] REPLICATION: finished
0506 11:38:46.437188 (+ 681us) write_op.cc:301] APPLY: starting
0506 11:38:46.437207 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:46.437247 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:46.437269 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:46.437390 (+ 121us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:46.437423 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:46.437430 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:46.437528 (+ 98us) log.cc:842] Serialized 22 byte log entry
0506 11:38:46.437646 (+ 118us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:46.437665 (+ 19us) write_op.cc:454] Released schema lock
0506 11:38:46.437673 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":605,"mutex_wait_us":288,"num_ops":1,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":662,"prepare.run_wall_time_us":886006,"raft.queue_time_us":322,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":879862,"spinlock_wait_cycles":14080,"thread_start_us":749,"threads_started":1}]]}
I20240506 11:38:46.439772 12012 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 31) took 1563 ms. Trace:
I20240506 11:38:46.440021 12012 rpcz_store.cc:276] 0506 11:38:44.876259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:44.876298 (+ 39us) service_pool.cc:225] Handling call
0506 11:38:44.876429 (+ 131us) tablet_service.cc:2780] Created scanner 58813b94f3d3460fbbce24c728d43dcc for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is f77885c72d1041e997b42c82b4acf5d7
0506 11:38:44.877370 (+ 941us) tablet_service.cc:2920] Creating iterator
0506 11:38:44.877414 (+ 44us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:45.556719 (+679305us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:46.437767 (+881048us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1560343 microseconds
0506 11:38:46.437853 (+ 86us) tablet_service.cc:2945] Iterator created
0506 11:38:46.438404 (+ 551us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:46.438434 (+ 30us) tablet_service.cc:3010] has_more: true
0506 11:38:46.438467 (+ 33us) tablet_service.cc:3027] Continuing scan request
0506 11:38:46.438558 (+ 91us) tablet_service.cc:3091] Found scanner 58813b94f3d3460fbbce24c728d43dcc for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is f77885c72d1041e997b42c82b4acf5d7
0506 11:38:46.439641 (+ 1083us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":34944}
I20240506 11:38:46.440397 12011 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 30) took 1564 ms. Trace:
I20240506 11:38:46.440588 12011 rpcz_store.cc:276] 0506 11:38:44.876095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:44.876170 (+ 75us) service_pool.cc:225] Handling call
0506 11:38:44.876384 (+ 214us) tablet_service.cc:2780] Created scanner 5467f093442a4127a018e0fe877f042b for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 4069a8bbeef34431b00d8ce5a9eb821a
0506 11:38:44.877373 (+ 989us) tablet_service.cc:2920] Creating iterator
0506 11:38:44.877410 (+ 37us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:45.556708 (+679298us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:46.437685 (+880977us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1560253 microseconds
0506 11:38:46.437833 (+ 148us) tablet_service.cc:2945] Iterator created
0506 11:38:46.438525 (+ 692us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:46.438562 (+ 37us) tablet_service.cc:3010] has_more: true
0506 11:38:46.438609 (+ 47us) tablet_service.cc:3027] Continuing scan request
0506 11:38:46.438712 (+ 103us) tablet_service.cc:3091] Found scanner 5467f093442a4127a018e0fe877f042b for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 4069a8bbeef34431b00d8ce5a9eb821a
0506 11:38:46.440245 (+ 1533us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":11776}
W20240506 11:38:46.441833 12403 scanner-internal.cc:458] Time spent opening tablet: real 1.566s user 0.001s sys 0.000s
W20240506 11:38:46.441833 12404 scanner-internal.cc:458] Time spent opening tablet: real 1.566s user 0.000s sys 0.000s
I20240506 11:38:46.507892 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:46.508090 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:46.508893 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:46.509420 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:46.509534 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:46.510138 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:46.587103 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:46.588681 12480 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=13, attempt_no=0}) took 1863 ms. Trace:
I20240506 11:38:46.588788 12480 rpcz_store.cc:276] 0506 11:38:44.725429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:44.725506 (+ 77us) service_pool.cc:225] Handling call
0506 11:38:46.588632 (+1863126us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:45.557254 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:45.557374 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:45.557379 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:38:45.557381 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:45.557468 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:45.557479 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:45.557483 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:45.557531 (+ 48us) tablet.cc:699] Row locks acquired
0506 11:38:45.557533 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:45.557570 (+ 37us) write_op.cc:270] Start()
0506 11:38:46.064707 (+507137us) write_op.cc:276] Timestamp: P: 1714995525557538 usec, L: 0
0506 11:38:46.064712 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:38:46.064980 (+ 268us) log.cc:842] Serialized 354 byte log entry
0506 11:38:46.588042 (+523062us) op_driver.cc:444] REPLICATION: finished
0506 11:38:46.588157 (+ 115us) write_op.cc:301] APPLY: starting
0506 11:38:46.588177 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:46.588213 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:46.588235 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:46.588303 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:46.588340 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:46.588346 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:38:46.588422 (+ 76us) log.cc:842] Serialized 22 byte log entry
0506 11:38:46.588501 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:46.588520 (+ 19us) write_op.cc:454] Released schema lock
0506 11:38:46.588548 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":50,"num_ops":1,"prepare.queue_time_us":831415,"prepare.run_cpu_time_us":749,"prepare.run_wall_time_us":507808,"replication_time_us":523306}]]}
I20240506 11:38:47.248476 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:47.248698 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:47.249536 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:47.250030 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:47.250232 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 17 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:47.250883 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:47.623098 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:47.627081 12488 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=14, attempt_no=0}) took 1184 ms. Trace:
I20240506 11:38:47.627365 12011 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 34) took 1034 ms. Trace:
I20240506 11:38:47.627444 12489 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=15, attempt_no=0}) took 1036 ms. Trace:
I20240506 11:38:47.627460 12011 rpcz_store.cc:276] 0506 11:38:46.593072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:46.593131 (+ 59us) service_pool.cc:225] Handling call
0506 11:38:46.593327 (+ 196us) tablet_service.cc:2780] Created scanner 27d75e6f73944e1482af4f311dbc989f for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 2702057b98414b86b97a97242ef64b6f
0506 11:38:46.594208 (+ 881us) tablet_service.cc:2920] Creating iterator
0506 11:38:46.594239 (+ 31us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:47.038327 (+444088us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:47.625336 (+587009us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1031083 microseconds
0506 11:38:47.625441 (+ 105us) tablet_service.cc:2945] Iterator created
0506 11:38:47.626080 (+ 639us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:47.626116 (+ 36us) tablet_service.cc:3010] has_more: true
0506 11:38:47.626161 (+ 45us) tablet_service.cc:3027] Continuing scan request
0506 11:38:47.626287 (+ 126us) tablet_service.cc:3091] Found scanner 27d75e6f73944e1482af4f311dbc989f for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 2702057b98414b86b97a97242ef64b6f
0506 11:38:47.627307 (+ 1020us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20240506 11:38:47.627286 12488 rpcz_store.cc:276] 0506 11:38:46.442367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:46.442426 (+ 59us) service_pool.cc:225] Handling call
0506 11:38:47.627024 (+1184598us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:46.442832 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:46.442906 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:46.442910 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:38:46.442911 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:46.443000 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:46.443008 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:46.443011 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:46.443044 (+ 33us) tablet.cc:699] Row locks acquired
0506 11:38:46.443046 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:46.443078 (+ 32us) write_op.cc:270] Start()
0506 11:38:47.038201 (+595123us) write_op.cc:276] Timestamp: P: 1714995526443053 usec, L: 0
0506 11:38:47.038205 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:38:47.038473 (+ 268us) log.cc:842] Serialized 354 byte log entry
0506 11:38:47.624284 (+585811us) op_driver.cc:444] REPLICATION: finished
0506 11:38:47.624694 (+ 410us) write_op.cc:301] APPLY: starting
0506 11:38:47.624713 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:47.624762 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:47.624791 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:47.625029 (+ 238us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:47.625064 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:47.625071 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:47.625174 (+ 103us) log.cc:842] Serialized 22 byte log entry
0506 11:38:47.626865 (+ 1691us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:47.626885 (+ 20us) write_op.cc:454] Released schema lock
0506 11:38:47.626892 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":348,"num_ops":1,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":639,"prepare.run_wall_time_us":595702,"replication_time_us":586056,"thread_start_us":240,"threads_started":1}]]}
I20240506 11:38:47.628140 12012 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 35) took 1034 ms. Trace:
I20240506 11:38:47.627605 12489 rpcz_store.cc:276] 0506 11:38:46.590510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:46.590586 (+ 76us) service_pool.cc:225] Handling call
0506 11:38:47.627387 (+1036801us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:47.038570 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:47.038689 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:47.038696 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:38:47.038700 (+ 4us) tablet.cc:660] Decoding operations
0506 11:38:47.038788 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:47.038799 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:47.038803 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:47.038854 (+ 51us) tablet.cc:699] Row locks acquired
0506 11:38:47.038856 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:47.038896 (+ 40us) write_op.cc:270] Start()
0506 11:38:47.139056 (+100160us) write_op.cc:276] Timestamp: P: 1714995527038869 usec, L: 0
0506 11:38:47.139061 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:38:47.139307 (+ 246us) log.cc:842] Serialized 354 byte log entry
0506 11:38:47.624608 (+485301us) op_driver.cc:444] REPLICATION: finished
0506 11:38:47.625049 (+ 441us) write_op.cc:301] APPLY: starting
0506 11:38:47.625061 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:47.625089 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:47.625152 (+ 63us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:47.625232 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:47.625256 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:47.625262 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:38:47.625343 (+ 81us) log.cc:842] Serialized 22 byte log entry
0506 11:38:47.627217 (+ 1874us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x6190000b0158. stack: 00007f4bf0d1f756 00007f4bf0d1f510 00007f4befd5248d 0000000000955804 00000000009557cd 000000000095519a 00007f4bfda013ed 00007f4bfda062e9 00007f4bfd8f79fd 00007f4bfd8fdf56 00007f4bfd8fc2bc 00007f4bfd8d2d7e 00007f4bfd8d1098 00007f4bfd8d3e2b 00007f4bfd8d3ca8 0000000000a60bcc
0506 11:38:47.627226 (+ 9us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:47.627257 (+ 31us) write_op.cc:454] Released schema lock
0506 11:38:47.627265 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":368,"num_ops":1,"prepare.queue_time_us":447533,"prepare.run_cpu_time_us":733,"prepare.run_wall_time_us":100806,"replication_time_us":485527,"spinlock_wait_cycles":3388288,"thread_start_us":274,"threads_started":1}]]}
I20240506 11:38:47.628239 12012 rpcz_store.cc:276] 0506 11:38:46.593206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:46.593250 (+ 44us) service_pool.cc:225] Handling call
0506 11:38:46.593461 (+ 211us) tablet_service.cc:2780] Created scanner 522f6d5b7b5d4feb891fc16da45fdee0 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is d28c0cd2f710439a86c41df4f1789216
0506 11:38:46.594298 (+ 837us) tablet_service.cc:2920] Creating iterator
0506 11:38:46.594326 (+ 28us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:47.038350 (+444024us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:47.626180 (+587830us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1031840 microseconds
0506 11:38:47.626279 (+ 99us) tablet_service.cc:2945] Iterator created
0506 11:38:47.626724 (+ 445us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:47.626751 (+ 27us) tablet_service.cc:3010] has_more: true
0506 11:38:47.626780 (+ 29us) tablet_service.cc:3027] Continuing scan request
0506 11:38:47.626827 (+ 47us) tablet_service.cc:3091] Found scanner 522f6d5b7b5d4feb891fc16da45fdee0 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is d28c0cd2f710439a86c41df4f1789216
0506 11:38:47.628071 (+ 1244us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20240506 11:38:47.628846 12404 scanner-internal.cc:458] Time spent opening tablet: real 1.036s user 0.000s sys 0.000s
W20240506 11:38:47.629362 12403 scanner-internal.cc:458] Time spent opening tablet: real 1.037s user 0.001s sys 0.000s
I20240506 11:38:47.896636 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:47.896835 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:47.897524 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:47.898164 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:47.898191 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:47.899912 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:47.900372 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:47.998304 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:47.998492 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:47.999174 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:47.999670 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:47.999852 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:48.000315 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:48.309145 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:48.309361 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:48.310066 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:48.310583 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 21 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:48.310709 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 21 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:48.312471 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:48.313102 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:49.179355 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:49.179558 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:49.180384 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:49.180866 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 22 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:49.180904 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 22 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:49.181529 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:49.181792 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:49.182971 12499 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=18, attempt_no=0}) took 1283 ms. Trace:
I20240506 11:38:49.183130 12499 rpcz_store.cc:276] 0506 11:38:47.899911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:47.900231 (+ 320us) service_pool.cc:225] Handling call
0506 11:38:49.182897 (+1282666us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:48.217170 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:48.217289 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:48.217294 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:38:48.217296 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:48.217369 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:48.217378 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:48.217381 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:48.217437 (+ 56us) tablet.cc:699] Row locks acquired
0506 11:38:48.217441 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:38:48.217487 (+ 46us) write_op.cc:270] Start()
0506 11:38:48.634626 (+417139us) write_op.cc:276] Timestamp: P: 1714995528217442 usec, L: 0
0506 11:38:48.634632 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:38:48.634964 (+ 332us) log.cc:842] Serialized 354 byte log entry
0506 11:38:49.181068 (+546104us) op_driver.cc:444] REPLICATION: finished
0506 11:38:49.182116 (+ 1048us) write_op.cc:301] APPLY: starting
0506 11:38:49.182136 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:49.182181 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:49.182245 (+ 64us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:49.182359 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:49.182396 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:49.182403 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:49.182488 (+ 85us) log.cc:842] Serialized 22 byte log entry
0506 11:38:49.182788 (+ 300us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:49.182805 (+ 17us) write_op.cc:454] Released schema lock
0506 11:38:49.182812 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":960,"num_ops":1,"prepare.queue_time_us":316482,"prepare.run_cpu_time_us":803,"prepare.run_wall_time_us":417871,"replication_time_us":546406,"thread_start_us":857,"threads_started":1}]]}
W20240506 11:38:49.185375 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.721s user 0.000s sys 0.001s
W20240506 11:38:49.185343 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.721s user 0.001s sys 0.000s
I20240506 11:38:49.483407 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:49.483572 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:49.484383 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:49.484907 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 23 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:49.484925 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 23 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:49.485421 12499 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=19, attempt_no=0}) took 1172 ms. Trace:
I20240506 11:38:49.485491 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:49.485767 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:49.485531 12499 rpcz_store.cc:276] 0506 11:38:48.312960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:48.313313 (+ 353us) service_pool.cc:225] Handling call
0506 11:38:49.485368 (+1172055us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:48.635086 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:48.635190 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:48.635196 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:48.635199 (+ 3us) tablet.cc:660] Decoding operations
0506 11:38:48.635296 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:48.635308 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:48.635313 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:48.635381 (+ 68us) tablet.cc:699] Row locks acquired
0506 11:38:48.635384 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:38:48.635424 (+ 40us) write_op.cc:270] Start()
0506 11:38:48.864571 (+229147us) write_op.cc:276] Timestamp: P: 1714995528635388 usec, L: 0
0506 11:38:48.864576 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:38:48.864801 (+ 225us) log.cc:842] Serialized 354 byte log entry
0506 11:38:49.484243 (+619442us) op_driver.cc:444] REPLICATION: finished
0506 11:38:49.484411 (+ 168us) write_op.cc:301] APPLY: starting
0506 11:38:49.484430 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:49.484472 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:49.484512 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:49.484586 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:49.484625 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:49.484632 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:49.484714 (+ 82us) log.cc:842] Serialized 22 byte log entry
0506 11:38:49.485240 (+ 526us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:49.485261 (+ 21us) write_op.cc:454] Released schema lock
0506 11:38:49.485269 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":107,"num_ops":1,"prepare.queue_time_us":321303,"prepare.run_cpu_time_us":730,"prepare.run_wall_time_us":229791,"replication_time_us":619640}]]}
I20240506 11:38:49.865840 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:49.865871 12182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:43381 (request call id 2140) took 1866 ms. Trace:
I20240506 11:38:49.866034 12182 rpcz_store.cc:276] 0506 11:38:47.999791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:48.000275 (+ 484us) service_pool.cc:225] Handling call
0506 11:38:48.000501 (+ 226us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 11:38:48.000632 (+ 131us) raft_consensus.cc:1520] Early marking committed up to index 18
0506 11:38:48.000956 (+ 324us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 11:38:48.001051 (+ 95us) write_op.cc:270] Start()
0506 11:38:48.947228 (+946177us) write_op.cc:276] Timestamp: P: 1714995527038869 usec, L: 0
0506 11:38:48.947823 (+ 595us) write_op.cc:270] Start()
0506 11:38:49.863985 (+916162us) write_op.cc:276] Timestamp: P: 1714995527630669 usec, L: 0
0506 11:38:49.864716 (+ 731us) log.cc:842] Serialized 708 byte log entry
0506 11:38:49.865057 (+ 341us) raft_consensus.cc:1631] Marking committed up to 20
0506 11:38:49.865713 (+ 656us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:38:49.865745 (+ 32us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:38:49.865757 (+ 12us) raft_consensus.cc:1671] finished
0506 11:38:49.865760 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:38:49.865830 (+ 70us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:48.947728 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:48.947850 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:48.947855 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:38:48.947857 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:48.947934 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:48.947947 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:48.947949 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:48.948004 (+ 55us) tablet.cc:699] Row locks acquired
0506 11:38:48.948006 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:49.865101 (+917095us) op_driver.cc:444] REPLICATION: finished
0506 11:38:49.865491 (+ 390us) write_op.cc:301] APPLY: starting
0506 11:38:49.865508 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:49.865554 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:49.865558 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:49.865703 (+ 145us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:49.865739 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:49.865824 (+ 85us) write_op.cc:312] APPLY: finished
0506 11:38:49.865953 (+ 129us) log.cc:842] Serialized 22 byte log entry
0506 11:38:49.866021 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:49.866040 (+ 19us) write_op.cc:454] Released schema lock
0506 11:38:49.866051 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:49.866151 (+ 100us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:49.866155 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:49.864568 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:49.864673 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:49.864679 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:49.864681 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:49.864769 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:49.864782 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:49.864784 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:49.864861 (+ 77us) tablet.cc:699] Row locks acquired
0506 11:38:49.864862 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:49.865406 (+ 544us) op_driver.cc:444] REPLICATION: finished
0506 11:38:49.865847 (+ 441us) write_op.cc:301] APPLY: starting
0506 11:38:49.865861 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:49.865889 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:49.865893 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:49.865977 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:49.866001 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:49.866067 (+ 66us) write_op.cc:312] APPLY: finished
0506 11:38:49.866159 (+ 92us) log.cc:842] Serialized 22 byte log entry
0506 11:38:49.866231 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:49.866246 (+ 15us) write_op.cc:454] Released schema lock
0506 11:38:49.866252 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:49.866349 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:49.866355 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"thread_start_us":257,"threads_started":1,"wal-append.queue_time_us":357,"child_traces":[["op",{"apply.queue_time_us":308,"apply.run_cpu_time_us":725,"apply.run_wall_time_us":724,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":360,"prepare.run_cpu_time_us":378,"prepare.run_wall_time_us":379,"replication_time_us":1864032,"thread_start_us":468,"threads_started":2}],["op",{"apply.queue_time_us":371,"apply.run_cpu_time_us":559,"apply.run_wall_time_us":559,"mutex_wait_us":5,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":451,"prepare.run_cpu_time_us":379,"prepare.run_wall_time_us":380,"replication_time_us":917570,"thread_start_us":578,"threads_started":2}]]}
I20240506 11:38:50.159157 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:50.159368 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:50.160063 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:50.160589 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 24 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:50.160619 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 24 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:50.161237 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
W20240506 11:38:50.163516 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.827s user 0.001s sys 0.000s
W20240506 11:38:50.163985 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.828s user 0.000s sys 0.001s
I20240506 11:38:50.381496 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:51.294994 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:51.295200 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:51.295943 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:51.296377 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 26 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:51.296494 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 26 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:51.297116 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:51.297650 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:51.298265 12511 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=22, attempt_no=0}) took 1134 ms. Trace:
I20240506 11:38:51.298444 12511 rpcz_store.cc:276] 0506 11:38:50.163643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:50.163712 (+ 69us) service_pool.cc:225] Handling call
0506 11:38:51.298202 (+1134490us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:50.164181 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:50.164249 (+ 68us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:50.164252 (+ 3us) write_op.cc:435] Acquired schema lock
0506 11:38:50.164254 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:50.164310 (+ 56us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:50.164318 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:50.164321 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:50.164365 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:38:50.164366 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:50.164393 (+ 27us) write_op.cc:270] Start()
0506 11:38:50.855521 (+691128us) write_op.cc:276] Timestamp: P: 1714995530164368 usec, L: 0
0506 11:38:50.855526 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:38:50.855778 (+ 252us) log.cc:842] Serialized 354 byte log entry
0506 11:38:51.296104 (+440326us) op_driver.cc:444] REPLICATION: finished
0506 11:38:51.296628 (+ 524us) write_op.cc:301] APPLY: starting
0506 11:38:51.296647 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:51.296686 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:51.296716 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:51.297074 (+ 358us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:51.297110 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:51.297117 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:51.297205 (+ 88us) log.cc:842] Serialized 22 byte log entry
0506 11:38:51.298070 (+ 865us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:51.298087 (+ 17us) write_op.cc:454] Released schema lock
0506 11:38:51.298094 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":455,"num_ops":1,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":598,"prepare.run_wall_time_us":691661,"replication_time_us":440553,"thread_start_us":748,"threads_started":1}]]}
W20240506 11:38:51.299882 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.986s user 0.001s sys 0.000s
W20240506 11:38:51.299882 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.985s user 0.000s sys 0.000s
I20240506 11:38:52.035491 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:52.035672 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:52.036401 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:52.036794 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 27 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:52.037067 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 27 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:52.037572 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:52.037842 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:52.039718 12515 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=23, attempt_no=0}) took 1655 ms. Trace:
W20240506 11:38:52.040145 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.590s user 0.001s sys 0.000s
W20240506 11:38:52.040145 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.590s user 0.001s sys 0.000s
I20240506 11:38:52.039816 12515 rpcz_store.cc:276] 0506 11:38:50.384489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:50.384548 (+ 59us) service_pool.cc:225] Handling call
0506 11:38:52.039660 (+1655112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:50.855885 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:50.855998 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:50.856004 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:50.856006 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:50.856114 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:50.856126 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:50.856129 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:50.856183 (+ 54us) tablet.cc:699] Row locks acquired
0506 11:38:50.856184 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:50.856218 (+ 34us) write_op.cc:270] Start()
0506 11:38:51.724357 (+868139us) write_op.cc:276] Timestamp: P: 1714995530856168 usec, L: 0
0506 11:38:51.724362 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:38:51.724574 (+ 212us) log.cc:842] Serialized 354 byte log entry
0506 11:38:52.037178 (+312604us) op_driver.cc:444] REPLICATION: finished
0506 11:38:52.037695 (+ 517us) write_op.cc:301] APPLY: starting
0506 11:38:52.037712 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:52.037752 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:52.037782 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:52.037847 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:52.037878 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:52.037885 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:52.037970 (+ 85us) log.cc:842] Serialized 22 byte log entry
0506 11:38:52.039555 (+ 1585us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:52.039569 (+ 14us) write_op.cc:454] Released schema lock
0506 11:38:52.039577 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":452,"num_ops":1,"prepare.queue_time_us":470936,"prepare.run_cpu_time_us":722,"prepare.run_wall_time_us":868794,"raft.queue_time_us":63,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":184,"replication_time_us":312792,"thread_start_us":873,"threads_started":1}]]}
I20240506 11:38:52.985918 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:52.986099 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:52.987056 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:52.987800 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 28 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:52.987944 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 28 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:52.988827 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:52.989145 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:52.990845 12522 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=24, attempt_no=0}) took 1690 ms. Trace:
I20240506 11:38:52.990990 12522 rpcz_store.cc:276] 0506 11:38:51.300426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:51.300482 (+ 56us) service_pool.cc:225] Handling call
0506 11:38:52.990796 (+1690314us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:51.724715 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:51.724836 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:51.724842 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:51.724844 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:51.724948 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:51.724963 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:51.724969 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:51.725022 (+ 53us) tablet.cc:699] Row locks acquired
0506 11:38:51.725025 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:38:51.725063 (+ 38us) write_op.cc:270] Start()
0506 11:38:52.333212 (+608149us) write_op.cc:276] Timestamp: P: 1714995531725031 usec, L: 0
0506 11:38:52.333220 (+ 8us) op_driver.cc:331] REPLICATION: starting
0506 11:38:52.333590 (+ 370us) log.cc:842] Serialized 354 byte log entry
0506 11:38:52.988889 (+655299us) op_driver.cc:444] REPLICATION: finished
0506 11:38:52.989557 (+ 668us) write_op.cc:301] APPLY: starting
0506 11:38:52.989572 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:52.989610 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:52.989628 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:52.989703 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:52.989726 (+ 23us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:52.989730 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:38:52.989834 (+ 104us) log.cc:842] Serialized 22 byte log entry
0506 11:38:52.990679 (+ 845us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:52.990694 (+ 15us) write_op.cc:454] Released schema lock
0506 11:38:52.990700 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":596,"num_ops":1,"prepare.queue_time_us":423911,"prepare.run_cpu_time_us":877,"prepare.run_wall_time_us":608954,"replication_time_us":655640,"thread_start_us":513,"threads_started":1}]]}
W20240506 11:38:52.992241 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.801s user 0.001s sys 0.000s
W20240506 11:38:52.992244 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.801s user 0.001s sys 0.000s
I20240506 11:38:53.225267 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:53.225461 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:53.226240 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:53.226718 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:53.226704 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 29 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:53.227154 12522 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=25, attempt_no=0}) took 1185 ms. Trace:
I20240506 11:38:53.227283 12522 rpcz_store.cc:276] 0506 11:38:52.041509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:52.041569 (+ 60us) service_pool.cc:225] Handling call
0506 11:38:53.227088 (+1185519us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:52.333721 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:52.333844 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:52.333850 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:52.333852 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:52.333951 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:52.333963 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:52.333967 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:52.334021 (+ 54us) tablet.cc:699] Row locks acquired
0506 11:38:52.334024 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:38:52.334061 (+ 37us) write_op.cc:270] Start()
0506 11:38:53.176224 (+842163us) write_op.cc:276] Timestamp: P: 1714995532334024 usec, L: 0
0506 11:38:53.176230 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:38:53.176494 (+ 264us) log.cc:842] Serialized 354 byte log entry
0506 11:38:53.226445 (+ 49951us) op_driver.cc:444] REPLICATION: finished
0506 11:38:53.226576 (+ 131us) write_op.cc:301] APPLY: starting
0506 11:38:53.226594 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:53.226636 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:53.226659 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:53.226734 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:53.226768 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:53.226774 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:38:53.226854 (+ 80us) log.cc:842] Serialized 22 byte log entry
0506 11:38:53.226966 (+ 112us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:53.226985 (+ 19us) write_op.cc:454] Released schema lock
0506 11:38:53.226993 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":77,"num_ops":1,"prepare.queue_time_us":291800,"prepare.run_cpu_time_us":800,"prepare.run_wall_time_us":842878,"raft.queue_time_us":58,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":167,"replication_time_us":50174,"spinlock_wait_cycles":363648}]]}
I20240506 11:38:53.229105 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:53.229405 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:53.734411 12182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:43381 (request call id 2141) took 3866 ms. Trace:
I20240506 11:38:53.734429 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:53.734691 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:53.735780 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:53.734592 12182 rpcz_store.cc:276] 0506 11:38:49.868308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:49.868366 (+ 58us) service_pool.cc:225] Handling call
0506 11:38:49.868584 (+ 218us) raft_consensus.cc:1457] Updating replica for 5 ops
0506 11:38:49.868731 (+ 147us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 11:38:49.868733 (+ 2us) raft_consensus.cc:1525] Triggering prepare for 5 ops
0506 11:38:49.868825 (+ 92us) write_op.cc:270] Start()
0506 11:38:50.844981 (+976156us) write_op.cc:276] Timestamp: P: 1714995527811502 usec, L: 0
0506 11:38:50.845421 (+ 440us) write_op.cc:270] Start()
0506 11:38:51.428600 (+583179us) write_op.cc:276] Timestamp: P: 1714995528217442 usec, L: 0
0506 11:38:51.429116 (+ 516us) write_op.cc:270] Start()
0506 11:38:52.308291 (+879175us) write_op.cc:276] Timestamp: P: 1714995528635388 usec, L: 0
0506 11:38:52.308881 (+ 590us) write_op.cc:270] Start()
0506 11:38:52.943036 (+634155us) write_op.cc:276] Timestamp: P: 1714995529186073 usec, L: 0
0506 11:38:52.943635 (+ 599us) write_op.cc:270] Start()
0506 11:38:53.731784 (+788149us) write_op.cc:276] Timestamp: P: 1714995529487812 usec, L: 0
0506 11:38:53.732498 (+ 714us) log.cc:842] Serialized 1770 byte log entry
0506 11:38:53.732796 (+ 298us) raft_consensus.cc:1631] Marking committed up to 23
0506 11:38:53.734234 (+ 1438us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:38:53.734269 (+ 35us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:38:53.734280 (+ 11us) raft_consensus.cc:1671] finished
0506 11:38:53.734283 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:38:53.734339 (+ 56us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:50.845388 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:50.845486 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:50.845491 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:38:50.845493 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:50.845585 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:50.845601 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:50.845605 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:50.845660 (+ 55us) tablet.cc:699] Row locks acquired
0506 11:38:50.845662 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:53.732841 (+2887179us) op_driver.cc:444] REPLICATION: finished
0506 11:38:53.733232 (+ 391us) write_op.cc:301] APPLY: starting
0506 11:38:53.733251 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:53.733439 (+ 188us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:53.733442 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:53.733518 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:53.733559 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:53.733593 (+ 34us) write_op.cc:312] APPLY: finished
0506 11:38:53.733670 (+ 77us) log.cc:842] Serialized 22 byte log entry
0506 11:38:53.733738 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:53.733756 (+ 18us) write_op.cc:454] Released schema lock
0506 11:38:53.733761 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:53.733858 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:53.733863 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:51.429053 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:51.429185 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:51.429193 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:38:51.429195 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:51.429286 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:51.429300 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:51.429308 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:51.429368 (+ 60us) tablet.cc:699] Row locks acquired
0506 11:38:51.429371 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:38:53.733132 (+2303761us) op_driver.cc:444] REPLICATION: finished
0506 11:38:53.733563 (+ 431us) write_op.cc:301] APPLY: starting
0506 11:38:53.733573 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:53.733593 (+ 20us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:53.733595 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:53.733663 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:53.733682 (+ 19us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:53.733737 (+ 55us) write_op.cc:312] APPLY: finished
0506 11:38:53.733802 (+ 65us) log.cc:842] Serialized 22 byte log entry
0506 11:38:53.733894 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:53.733905 (+ 11us) write_op.cc:454] Released schema lock
0506 11:38:53.733909 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:53.733969 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:53.733973 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:52.308770 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:52.308898 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:52.308906 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:38:52.308910 (+ 4us) tablet.cc:660] Decoding operations
0506 11:38:52.308992 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:52.309002 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:52.309005 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:52.309050 (+ 45us) tablet.cc:699] Row locks acquired
0506 11:38:52.309052 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:53.733426 (+1424374us) op_driver.cc:444] REPLICATION: finished
0506 11:38:53.733952 (+ 526us) write_op.cc:301] APPLY: starting
0506 11:38:53.733962 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:53.733987 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:53.733989 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:53.734036 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:53.734071 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:53.734124 (+ 53us) write_op.cc:312] APPLY: finished
0506 11:38:53.734186 (+ 62us) log.cc:842] Serialized 22 byte log entry
0506 11:38:53.734245 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:53.734282 (+ 37us) write_op.cc:454] Released schema lock
0506 11:38:53.734289 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:53.734349 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:53.734353 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:52.943570 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:52.943672 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:52.943677 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:38:52.943678 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:52.943752 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:52.943763 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:52.943766 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:52.943816 (+ 50us) tablet.cc:699] Row locks acquired
0506 11:38:52.943818 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0506 11:38:53.732330 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:53.732454 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:53.732458 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:38:53.732460 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:53.732532 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:53.732544 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:53.732547 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:53.732591 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:38:53.732593 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":221,"threads_started":1,"wal-append.queue_time_us":320,"child_traces":[["op",{"apply.queue_time_us":321,"apply.run_cpu_time_us":596,"apply.run_wall_time_us":690,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":288,"prepare.run_cpu_time_us":367,"prepare.run_wall_time_us":368,"replication_time_us":3863996,"thread_start_us":403,"threads_started":2}],["op",{"apply.queue_time_us":348,"apply.run_cpu_time_us":447,"apply.run_wall_time_us":447,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":292,"prepare.run_cpu_time_us":450,"prepare.run_wall_time_us":450,"replication_time_us":2887696,"thread_start_us":417,"threads_started":2}],["op",{"apply.queue_time_us":487,"apply.run_cpu_time_us":480,"apply.run_wall_time_us":546,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":320,"prepare.run_cpu_time_us":365,"prepare.run_wall_time_us":365,"replication_time_us":2304297,"thread_start_us":938,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":406,"prepare.run_cpu_time_us":309,"prepare.run_wall_time_us":310,"thread_start_us":300,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":409,"prepare.run_cpu_time_us":329,"prepare.run_wall_time_us":330,"thread_start_us":220,"threads_started":1}]]}
I20240506 11:38:53.736402 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:53.736402 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 25 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:53.736979 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:53.737208 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:54.406572 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:54.406750 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:54.407657 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:54.408157 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 30 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:54.408175 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 30 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:54.408823 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:54.409122 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:54.409478 12533 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=26, attempt_no=0}) took 1416 ms. Trace:
I20240506 11:38:54.409719 12533 rpcz_store.cc:276] 0506 11:38:52.992847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:52.992912 (+ 65us) service_pool.cc:225] Handling call
0506 11:38:54.409422 (+1416510us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:53.176629 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:53.176735 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:53.176740 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:38:53.176742 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:53.176835 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:53.176845 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:53.176847 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:53.176890 (+ 43us) tablet.cc:699] Row locks acquired
0506 11:38:53.176892 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:53.176933 (+ 41us) write_op.cc:270] Start()
0506 11:38:53.654077 (+477144us) write_op.cc:276] Timestamp: P: 1714995533176896 usec, L: 0
0506 11:38:53.654081 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:38:53.654359 (+ 278us) log.cc:842] Serialized 354 byte log entry
0506 11:38:54.408249 (+753890us) op_driver.cc:444] REPLICATION: finished
0506 11:38:54.408830 (+ 581us) write_op.cc:301] APPLY: starting
0506 11:38:54.408847 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:54.408895 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:54.408922 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:54.409035 (+ 113us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:54.409071 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:54.409078 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:54.409171 (+ 93us) log.cc:842] Serialized 22 byte log entry
0506 11:38:54.409285 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:54.409303 (+ 18us) write_op.cc:454] Released schema lock
0506 11:38:54.409310 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":507,"num_ops":1,"prepare.queue_time_us":183397,"prepare.run_cpu_time_us":801,"prepare.run_wall_time_us":477875,"raft.queue_time_us":23,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":186,"replication_time_us":754144,"spinlock_wait_cycles":475776,"thread_start_us":777,"threads_started":1}]]}
I20240506 11:38:54.411146 12010 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 65) took 1030 ms. Trace:
I20240506 11:38:54.411245 12010 rpcz_store.cc:276] 0506 11:38:53.381057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:53.381105 (+ 48us) service_pool.cc:225] Handling call
0506 11:38:53.381293 (+ 188us) tablet_service.cc:2780] Created scanner cf6295ce4a5e4a80bf9b06a6cd424ca3 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 392c652e8970465081a5101f009887c4
0506 11:38:53.382058 (+ 765us) tablet_service.cc:2920] Creating iterator
0506 11:38:53.382080 (+ 22us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:53.654227 (+272147us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:54.409307 (+755080us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1027213 microseconds
0506 11:38:54.409406 (+ 99us) tablet_service.cc:2945] Iterator created
0506 11:38:54.409995 (+ 589us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:54.410038 (+ 43us) tablet_service.cc:3010] has_more: true
0506 11:38:54.410079 (+ 41us) tablet_service.cc:3027] Continuing scan request
0506 11:38:54.410159 (+ 80us) tablet_service.cc:3091] Found scanner cf6295ce4a5e4a80bf9b06a6cd424ca3 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 392c652e8970465081a5101f009887c4
0506 11:38:54.411078 (+ 919us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20240506 11:38:54.411504 12012 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 64) took 1031 ms. Trace:
I20240506 11:38:54.411617 12012 rpcz_store.cc:276] 0506 11:38:53.380288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:53.380349 (+ 61us) service_pool.cc:225] Handling call
0506 11:38:53.380557 (+ 208us) tablet_service.cc:2780] Created scanner 83aac1a334d44686ad0646628eee351f for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is f2a5469df6ed4711b3e4cc8388757680
0506 11:38:53.381374 (+ 817us) tablet_service.cc:2920] Creating iterator
0506 11:38:53.381410 (+ 36us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:53.654224 (+272814us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:54.409470 (+755246us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1028045 microseconds
0506 11:38:54.409584 (+ 114us) tablet_service.cc:2945] Iterator created
0506 11:38:54.410337 (+ 753us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:54.410376 (+ 39us) tablet_service.cc:3010] has_more: true
0506 11:38:54.410421 (+ 45us) tablet_service.cc:3027] Continuing scan request
0506 11:38:54.410503 (+ 82us) tablet_service.cc:3091] Found scanner 83aac1a334d44686ad0646628eee351f for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is f2a5469df6ed4711b3e4cc8388757680
0506 11:38:54.411408 (+ 905us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":4864}
W20240506 11:38:54.412009 12403 scanner-internal.cc:458] Time spent opening tablet: real 1.031s user 0.001s sys 0.000s
W20240506 11:38:54.412452 12404 scanner-internal.cc:458] Time spent opening tablet: real 1.033s user 0.001s sys 0.000s
I20240506 11:38:54.902812 12383 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:54.903066 12383 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:54.904075 12383 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:54.904500 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 31 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:54.904594 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 31 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:54.904891 12533 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=27, attempt_no=0}) took 1675 ms. Trace:
I20240506 11:38:54.905257 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:54.904997 12533 rpcz_store.cc:276] 0506 11:38:53.229179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:53.229493 (+ 314us) service_pool.cc:225] Handling call
0506 11:38:54.904842 (+1675349us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:53.654524 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:53.654641 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:53.654646 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:38:53.654647 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:53.654740 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:53.654752 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:53.654757 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:53.654812 (+ 55us) tablet.cc:699] Row locks acquired
0506 11:38:53.654814 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:53.654845 (+ 31us) write_op.cc:270] Start()
0506 11:38:53.992031 (+337186us) write_op.cc:276] Timestamp: P: 1714995533654809 usec, L: 0
0506 11:38:53.992040 (+ 9us) op_driver.cc:331] REPLICATION: starting
0506 11:38:53.992258 (+ 218us) log.cc:842] Serialized 354 byte log entry
0506 11:38:54.903953 (+911695us) op_driver.cc:444] REPLICATION: finished
0506 11:38:54.904293 (+ 340us) write_op.cc:301] APPLY: starting
0506 11:38:54.904314 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:54.904358 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:54.904410 (+ 52us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:54.904487 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:54.904539 (+ 52us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:54.904546 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:54.904631 (+ 85us) log.cc:842] Serialized 22 byte log entry
0506 11:38:54.904718 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:54.904735 (+ 17us) write_op.cc:454] Released schema lock
0506 11:38:54.904742 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":267,"num_ops":1,"prepare.queue_time_us":424625,"prepare.run_cpu_time_us":744,"prepare.run_wall_time_us":337805,"replication_time_us":911883}]]}
I20240506 11:38:55.099040 12383 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 11:38:55.102753 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.540s user 0.000s sys 0.001s
W20240506 11:38:55.102787 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.540s user 0.000s sys 0.000s
I20240506 11:38:55.263998 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:55.264164 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:55.264801 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:55.265028 12182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:43381 (request call id 2142) took 1527 ms. Trace:
I20240506 11:38:55.265089 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 30 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:55.265233 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 30 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:55.265130 12182 rpcz_store.cc:276] 0506 11:38:53.737504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:53.737592 (+ 88us) service_pool.cc:225] Handling call
0506 11:38:53.737787 (+ 195us) raft_consensus.cc:1457] Updating replica for 5 ops
0506 11:38:53.737930 (+ 143us) raft_consensus.cc:1520] Early marking committed up to index 25
0506 11:38:53.738088 (+ 158us) raft_consensus.cc:1525] Triggering prepare for 5 ops
0506 11:38:53.738199 (+ 111us) write_op.cc:270] Start()
0506 11:38:54.055424 (+317225us) write_op.cc:276] Timestamp: P: 1714995530164368 usec, L: 0
0506 11:38:54.055709 (+ 285us) write_op.cc:270] Start()
0506 11:38:54.153893 (+ 98184us) write_op.cc:276] Timestamp: P: 1714995530856168 usec, L: 0
0506 11:38:54.154206 (+ 313us) write_op.cc:270] Start()
0506 11:38:54.381368 (+227162us) write_op.cc:276] Timestamp: P: 1714995531725031 usec, L: 0
0506 11:38:54.381665 (+ 297us) write_op.cc:270] Start()
0506 11:38:55.053823 (+672158us) write_op.cc:276] Timestamp: P: 1714995532334024 usec, L: 0
0506 11:38:55.054356 (+ 533us) write_op.cc:270] Start()
0506 11:38:55.260522 (+206166us) write_op.cc:276] Timestamp: P: 1714995533176896 usec, L: 0
0506 11:38:55.260968 (+ 446us) log.cc:842] Serialized 1770 byte log entry
0506 11:38:55.261311 (+ 343us) raft_consensus.cc:1631] Marking committed up to 29
0506 11:38:55.263912 (+ 2601us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:38:55.264944 (+ 1032us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:38:55.264953 (+ 9us) raft_consensus.cc:1671] finished
0506 11:38:55.264955 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:38:55.264996 (+ 41us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:54.055630 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:54.055738 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:54.055743 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:38:54.055744 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:54.055818 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:54.055831 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:54.055834 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:54.055876 (+ 42us) tablet.cc:699] Row locks acquired
0506 11:38:54.055877 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:55.261350 (+1205473us) op_driver.cc:444] REPLICATION: finished
0506 11:38:55.261815 (+ 465us) write_op.cc:301] APPLY: starting
0506 11:38:55.261832 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:55.261887 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:55.261890 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:55.261979 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:55.262022 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:55.262066 (+ 44us) write_op.cc:312] APPLY: finished
0506 11:38:55.262141 (+ 75us) log.cc:842] Serialized 22 byte log entry
0506 11:38:55.262198 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:55.262218 (+ 20us) write_op.cc:454] Released schema lock
0506 11:38:55.262224 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:55.262331 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:55.262336 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:54.154080 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:54.154202 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:54.154208 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:54.154209 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:54.154282 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:54.154292 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:54.154295 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:54.154337 (+ 42us) tablet.cc:699] Row locks acquired
0506 11:38:54.154338 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:55.261596 (+1107258us) op_driver.cc:444] REPLICATION: finished
0506 11:38:55.262463 (+ 867us) write_op.cc:301] APPLY: starting
0506 11:38:55.262477 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:55.262517 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:55.262522 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:55.262606 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:55.262643 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:55.262715 (+ 72us) write_op.cc:312] APPLY: finished
0506 11:38:55.262811 (+ 96us) log.cc:842] Serialized 22 byte log entry
0506 11:38:55.262871 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:55.262888 (+ 17us) write_op.cc:454] Released schema lock
0506 11:38:55.262895 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:55.263035 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:55.263041 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:54.381591 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:54.381712 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:54.381728 (+ 16us) write_op.cc:435] Acquired schema lock
0506 11:38:54.381729 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:54.381803 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:54.381814 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:54.381816 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:54.381879 (+ 63us) tablet.cc:699] Row locks acquired
0506 11:38:54.381883 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:38:55.262064 (+880181us) op_driver.cc:444] REPLICATION: finished
0506 11:38:55.262719 (+ 655us) write_op.cc:301] APPLY: starting
0506 11:38:55.262732 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:55.262763 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:55.262766 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:55.262815 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:55.262840 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:55.263220 (+ 380us) write_op.cc:312] APPLY: finished
0506 11:38:55.263832 (+ 612us) log.cc:842] Serialized 22 byte log entry
0506 11:38:55.263960 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:55.263972 (+ 12us) write_op.cc:454] Released schema lock
0506 11:38:55.263978 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:55.264318 (+ 340us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:55.264321 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:55.054298 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:55.054434 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:55.054440 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:55.054443 (+ 3us) tablet.cc:660] Decoding operations
0506 11:38:55.054520 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:55.054532 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:55.054535 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:55.054579 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:38:55.054581 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:55.262383 (+207802us) op_driver.cc:444] REPLICATION: finished
0506 11:38:55.263132 (+ 749us) write_op.cc:301] APPLY: starting
0506 11:38:55.263145 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:55.263175 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:55.263180 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:55.263236 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:55.263270 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:55.263304 (+ 34us) write_op.cc:312] APPLY: finished
0506 11:38:55.263400 (+ 96us) log.cc:842] Serialized 22 byte log entry
0506 11:38:55.263458 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:55.263485 (+ 27us) write_op.cc:454] Released schema lock
0506 11:38:55.263491 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:55.263554 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:55.263560 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:55.260730 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:55.260833 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:55.260839 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:55.260841 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:55.260945 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:55.260958 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:55.260963 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:55.261023 (+ 60us) tablet.cc:699] Row locks acquired
0506 11:38:55.261025 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":261,"threads_started":1,"wal-append.queue_time_us":340,"child_traces":[["op",{"apply.queue_time_us":282,"apply.run_cpu_time_us":585,"apply.run_wall_time_us":586,"mutex_wait_us":1258,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":71,"prepare.run_cpu_time_us":359,"prepare.run_wall_time_us":360,"replication_time_us":1523133,"thread_start_us":172,"threads_started":1}],["op",{"apply.queue_time_us":810,"apply.run_cpu_time_us":634,"apply.run_wall_time_us":634,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":351,"prepare.run_wall_time_us":352,"replication_time_us":1205876,"thread_start_us":389,"threads_started":1}],["op",{"apply.queue_time_us":591,"apply.run_cpu_time_us":562,"apply.run_wall_time_us":1644,"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":70,"prepare.run_cpu_time_us":381,"prepare.run_wall_time_us":382,"replication_time_us":1107845,"thread_start_us":260,"threads_started":1}],["op",{"apply.queue_time_us":714,"apply.run_cpu_time_us":472,"apply.run_wall_time_us":471,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":341,"prepare.run_cpu_time_us":344,"prepare.run_wall_time_us":344,"replication_time_us":880708,"thread_start_us":1648,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":77,"prepare.run_cpu_time_us":382,"prepare.run_wall_time_us":378}]]}
I20240506 11:38:55.267356 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:55.267779 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:55.655411 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:55.655622 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:55.656385 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:55.656906 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:55.657032 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:55.658282 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:56.333168 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:56.335621 12550 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=30, attempt_no=0}) took 1233 ms. Trace:
I20240506 11:38:56.335757 12550 rpcz_store.cc:276] 0506 11:38:55.102328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:55.102394 (+ 66us) service_pool.cc:225] Handling call
0506 11:38:56.335570 (+1233176us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:55.600912 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:55.601050 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:55.601057 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:38:55.601059 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:55.601154 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:55.601165 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:55.601170 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:55.601226 (+ 56us) tablet.cc:699] Row locks acquired
0506 11:38:55.601228 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:55.601269 (+ 41us) write_op.cc:270] Start()
0506 11:38:55.654420 (+ 53151us) write_op.cc:276] Timestamp: P: 1714995535601207 usec, L: 0
0506 11:38:55.654431 (+ 11us) op_driver.cc:331] REPLICATION: starting
0506 11:38:55.654666 (+ 235us) log.cc:842] Serialized 354 byte log entry
0506 11:38:56.334563 (+679897us) op_driver.cc:444] REPLICATION: finished
0506 11:38:56.335131 (+ 568us) write_op.cc:301] APPLY: starting
0506 11:38:56.335147 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:56.335193 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:56.335235 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:56.335300 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:56.335324 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:56.335329 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:38:56.335390 (+ 61us) log.cc:842] Serialized 22 byte log entry
0506 11:38:56.335457 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:56.335472 (+ 15us) write_op.cc:454] Released schema lock
0506 11:38:56.335476 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":485,"num_ops":1,"prepare.queue_time_us":498152,"prepare.run_cpu_time_us":761,"prepare.run_wall_time_us":53832,"replication_time_us":680109,"thread_start_us":341,"threads_started":1}]]}
I20240506 11:38:56.529307 12182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:43381 (request call id 2143) took 1261 ms. Trace:
I20240506 11:38:56.529428 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:56.529605 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:56.529495 12182 rpcz_store.cc:276] 0506 11:38:55.267832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:55.268153 (+ 321us) service_pool.cc:225] Handling call
0506 11:38:55.268336 (+ 183us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 11:38:55.268507 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 30
0506 11:38:55.268585 (+ 78us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 11:38:55.268686 (+ 101us) write_op.cc:270] Start()
0506 11:38:56.050849 (+782163us) write_op.cc:276] Timestamp: P: 1714995533654809 usec, L: 0
0506 11:38:56.051487 (+ 638us) write_op.cc:270] Start()
0506 11:38:56.527655 (+476168us) write_op.cc:276] Timestamp: P: 1714995534412930 usec, L: 0
0506 11:38:56.528069 (+ 414us) log.cc:842] Serialized 708 byte log entry
0506 11:38:56.528395 (+ 326us) raft_consensus.cc:1631] Marking committed up to 32
0506 11:38:56.529152 (+ 757us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:38:56.529186 (+ 34us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:38:56.529195 (+ 9us) raft_consensus.cc:1671] finished
0506 11:38:56.529197 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:38:56.529252 (+ 55us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:56.051390 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:56.051513 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:56.051520 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:38:56.051524 (+ 4us) tablet.cc:660] Decoding operations
0506 11:38:56.051610 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:56.051622 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:56.051625 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:56.051688 (+ 63us) tablet.cc:699] Row locks acquired
0506 11:38:56.051690 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:56.528425 (+476735us) op_driver.cc:444] REPLICATION: finished
0506 11:38:56.528748 (+ 323us) write_op.cc:301] APPLY: starting
0506 11:38:56.528764 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:56.528811 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:56.528816 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:56.528908 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:56.528958 (+ 50us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:56.529015 (+ 57us) write_op.cc:312] APPLY: finished
0506 11:38:56.529149 (+ 134us) log.cc:842] Serialized 22 byte log entry
0506 11:38:56.529216 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:56.529228 (+ 12us) write_op.cc:454] Released schema lock
0506 11:38:56.529233 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:56.529295 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:56.529297 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:56.527866 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:56.528014 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:56.528020 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:56.528022 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:56.528126 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:56.528141 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:56.528146 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:56.528208 (+ 62us) tablet.cc:699] Row locks acquired
0506 11:38:56.528212 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:38:56.528696 (+ 484us) op_driver.cc:444] REPLICATION: finished
0506 11:38:56.529194 (+ 498us) write_op.cc:301] APPLY: starting
0506 11:38:56.529227 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:56.529257 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:56.529259 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:56.529324 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:56.529355 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:56.529415 (+ 60us) write_op.cc:312] APPLY: finished
0506 11:38:56.529502 (+ 87us) log.cc:842] Serialized 22 byte log entry
0506 11:38:56.529545 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:56.529559 (+ 14us) write_op.cc:454] Released schema lock
0506 11:38:56.529564 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:56.529630 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:56.529635 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"thread_start_us":228,"threads_started":1,"wal-append.queue_time_us":344,"child_traces":[["op",{"apply.queue_time_us":275,"apply.run_cpu_time_us":593,"apply.run_wall_time_us":594,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":410,"prepare.run_cpu_time_us":393,"prepare.run_wall_time_us":392,"replication_time_us":1259727,"thread_start_us":463,"threads_started":2}],["op",{"apply.queue_time_us":456,"apply.run_cpu_time_us":496,"apply.run_wall_time_us":497,"mutex_wait_us":23,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":483,"prepare.run_wall_time_us":483,"replication_time_us":477199,"spinlock_wait_cycles":22912,"thread_start_us":284,"threads_started":1}]]}
I20240506 11:38:56.530349 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:56.530849 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:56.531004 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:56.531519 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:56.531785 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:57.153856 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:57.154055 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:57.154780 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:57.155252 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 35 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:57.155323 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 35 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:57.155943 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:57.156212 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:57.156756 12558 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=31, attempt_no=0}) took 1496 ms. Trace:
I20240506 11:38:57.156906 12558 rpcz_store.cc:276] 0506 11:38:55.659934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:55.660022 (+ 88us) service_pool.cc:225] Handling call
0506 11:38:57.156687 (+1496665us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:55.660454 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:55.660529 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:55.660533 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:38:55.660534 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:55.660591 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:55.660599 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:55.660602 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:55.660635 (+ 33us) tablet.cc:699] Row locks acquired
0506 11:38:55.660637 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:55.660663 (+ 26us) write_op.cc:270] Start()
0506 11:38:56.035810 (+375147us) write_op.cc:276] Timestamp: P: 1714995535660647 usec, L: 0
0506 11:38:56.035815 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:38:56.036116 (+ 301us) log.cc:842] Serialized 354 byte log entry
0506 11:38:57.155469 (+1119353us) op_driver.cc:444] REPLICATION: finished
0506 11:38:57.156172 (+ 703us) write_op.cc:301] APPLY: starting
0506 11:38:57.156189 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:57.156231 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:57.156261 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:57.156368 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:57.156398 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:57.156403 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:38:57.156482 (+ 79us) log.cc:842] Serialized 22 byte log entry
0506 11:38:57.156577 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:57.156594 (+ 17us) write_op.cc:454] Released schema lock
0506 11:38:57.156601 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":638,"num_ops":1,"prepare.queue_time_us":73,"prepare.run_cpu_time_us":658,"prepare.run_wall_time_us":375728,"replication_time_us":1119630,"thread_start_us":536,"threads_started":1}]]}
I20240506 11:38:57.158231 12010 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 75) took 1347 ms. Trace:
I20240506 11:38:57.158336 12010 rpcz_store.cc:276] 0506 11:38:55.811155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:55.811219 (+ 64us) service_pool.cc:225] Handling call
0506 11:38:55.811383 (+ 164us) tablet_service.cc:2780] Created scanner deb9d296a0524b138b95d02f248dfe62 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 1cd306ad43624cef9d8cff5a5ad6c00b
0506 11:38:55.812086 (+ 703us) tablet_service.cc:2920] Creating iterator
0506 11:38:55.812135 (+ 49us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:56.035995 (+223860us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:57.156607 (+1120612us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1344457 microseconds
0506 11:38:57.156715 (+ 108us) tablet_service.cc:2945] Iterator created
0506 11:38:57.157239 (+ 524us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:57.157267 (+ 28us) tablet_service.cc:3010] has_more: true
0506 11:38:57.157303 (+ 36us) tablet_service.cc:3027] Continuing scan request
0506 11:38:57.157378 (+ 75us) tablet_service.cc:3091] Found scanner deb9d296a0524b138b95d02f248dfe62 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 1cd306ad43624cef9d8cff5a5ad6c00b
0506 11:38:57.158192 (+ 814us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":24832}
I20240506 11:38:57.158546 12012 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 74) took 1347 ms. Trace:
I20240506 11:38:57.158648 12012 rpcz_store.cc:276] 0506 11:38:55.811020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:55.811092 (+ 72us) service_pool.cc:225] Handling call
0506 11:38:55.811331 (+ 239us) tablet_service.cc:2780] Created scanner 7c3445b5d40e4a5f95077bb750f2074c for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is a8a3705239ff4f81b4577e73d345ef97
0506 11:38:55.812091 (+ 760us) tablet_service.cc:2920] Creating iterator
0506 11:38:55.812137 (+ 46us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:56.035990 (+223853us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:57.156618 (+1120628us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1344470 microseconds
0506 11:38:57.156721 (+ 103us) tablet_service.cc:2945] Iterator created
0506 11:38:57.157261 (+ 540us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:57.157290 (+ 29us) tablet_service.cc:3010] has_more: true
0506 11:38:57.157332 (+ 42us) tablet_service.cc:3027] Continuing scan request
0506 11:38:57.157416 (+ 84us) tablet_service.cc:3091] Found scanner 7c3445b5d40e4a5f95077bb750f2074c for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is a8a3705239ff4f81b4577e73d345ef97
0506 11:38:57.158487 (+ 1071us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":40064}
W20240506 11:38:57.158919 12404 scanner-internal.cc:458] Time spent opening tablet: real 1.349s user 0.001s sys 0.000s
W20240506 11:38:57.159219 12403 scanner-internal.cc:458] Time spent opening tablet: real 1.349s user 0.001s sys 0.000s
I20240506 11:38:57.777120 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:57.777304 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:57.778105 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:57.778571 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 36 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:57.778767 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 36 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:57.779371 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:57.779930 12565 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=32, attempt_no=0}) took 1442 ms. Trace:
I20240506 11:38:57.780042 12565 rpcz_store.cc:276] 0506 11:38:56.337584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:56.337673 (+ 89us) service_pool.cc:225] Handling call
0506 11:38:57.779856 (+1442183us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:56.338165 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:56.338263 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:56.338267 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:38:56.338268 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:56.338329 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:56.338357 (+ 28us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:56.338372 (+ 15us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:56.338427 (+ 55us) tablet.cc:699] Row locks acquired
0506 11:38:56.338428 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:56.338461 (+ 33us) write_op.cc:270] Start()
0506 11:38:56.559596 (+221135us) write_op.cc:276] Timestamp: P: 1714995536338426 usec, L: 0
0506 11:38:56.559600 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:38:56.559812 (+ 212us) log.cc:842] Serialized 354 byte log entry
0506 11:38:57.778724 (+1218912us) op_driver.cc:444] REPLICATION: finished
0506 11:38:57.779342 (+ 618us) write_op.cc:301] APPLY: starting
0506 11:38:57.779359 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:57.779399 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:57.779428 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:57.779496 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:57.779530 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:57.779537 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:57.779609 (+ 72us) log.cc:842] Serialized 22 byte log entry
0506 11:38:57.779693 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:57.779712 (+ 19us) write_op.cc:454] Released schema lock
0506 11:38:57.779721 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":553,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":650,"prepare.run_wall_time_us":221714,"replication_time_us":1219082,"thread_start_us":1359,"threads_started":1}]]}
I20240506 11:38:58.362382 12182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:43381 (request call id 2144) took 1830 ms. Trace:
I20240506 11:38:58.362469 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:58.362710 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:58.363381 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:58.362588 12182 rpcz_store.cc:276] 0506 11:38:56.531859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:56.531929 (+ 70us) service_pool.cc:225] Handling call
0506 11:38:56.532117 (+ 188us) raft_consensus.cc:1457] Updating replica for 3 ops
0506 11:38:56.532243 (+ 126us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 11:38:56.532245 (+ 2us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0506 11:38:56.532393 (+ 148us) write_op.cc:270] Start()
0506 11:38:57.176537 (+644144us) write_op.cc:276] Timestamp: P: 1714995534907324 usec, L: 0
0506 11:38:57.177083 (+ 546us) write_op.cc:270] Start()
0506 11:38:57.668195 (+491112us) write_op.cc:276] Timestamp: P: 1714995535601207 usec, L: 0
0506 11:38:57.668499 (+ 304us) write_op.cc:270] Start()
0506 11:38:58.360661 (+692162us) write_op.cc:276] Timestamp: P: 1714995535660647 usec, L: 0
0506 11:38:58.361219 (+ 558us) log.cc:842] Serialized 1062 byte log entry
0506 11:38:58.361533 (+ 314us) raft_consensus.cc:1631] Marking committed up to 34
0506 11:38:58.362208 (+ 675us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:38:58.362238 (+ 30us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:38:58.362247 (+ 9us) raft_consensus.cc:1671] finished
0506 11:38:58.362249 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:38:58.362324 (+ 75us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:57.177014 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:57.177134 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:57.177139 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:38:57.177140 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:57.177210 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:57.177221 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:57.177224 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:57.177264 (+ 40us) tablet.cc:699] Row locks acquired
0506 11:38:57.177265 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:58.361572 (+1184307us) op_driver.cc:444] REPLICATION: finished
0506 11:38:58.361866 (+ 294us) write_op.cc:301] APPLY: starting
0506 11:38:58.361878 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:58.361909 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:58.361910 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:58.362023 (+ 113us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:58.362067 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:58.362110 (+ 43us) write_op.cc:312] APPLY: finished
0506 11:38:58.362220 (+ 110us) log.cc:842] Serialized 22 byte log entry
0506 11:38:58.362276 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:58.362290 (+ 14us) write_op.cc:454] Released schema lock
0506 11:38:58.362296 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:58.362449 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:58.362459 (+ 10us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:57.668403 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:57.668518 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:57.668524 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:57.668525 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:57.668600 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:57.668612 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:57.668615 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:57.668660 (+ 45us) tablet.cc:699] Row locks acquired
0506 11:38:57.668661 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:58.361822 (+693161us) op_driver.cc:444] REPLICATION: finished
0506 11:38:58.362198 (+ 376us) write_op.cc:301] APPLY: starting
0506 11:38:58.362209 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:58.362234 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:58.362238 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:58.362280 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:58.362302 (+ 22us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:58.362343 (+ 41us) write_op.cc:312] APPLY: finished
0506 11:38:58.362438 (+ 95us) log.cc:842] Serialized 22 byte log entry
0506 11:38:58.362488 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:58.362498 (+ 10us) write_op.cc:454] Released schema lock
0506 11:38:58.362503 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0506 11:38:58.362553 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:58.362556 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:38:58.361091 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:58.361196 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:58.361200 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:38:58.361202 (+ 2us) tablet.cc:660] Decoding operations
0506 11:38:58.361297 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:58.361312 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:58.361316 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:58.361372 (+ 56us) tablet.cc:699] Row locks acquired
0506 11:38:58.361374 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":166784,"thread_start_us":241,"threads_started":1,"wal-append.queue_time_us":309,"child_traces":[["op",{"apply.queue_time_us":237,"apply.run_cpu_time_us":589,"apply.run_wall_time_us":647,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":355,"prepare.run_cpu_time_us":338,"prepare.run_wall_time_us":338,"replication_time_us":1829163,"thread_start_us":435,"threads_started":2}],["op",{"apply.queue_time_us":317,"apply.run_cpu_time_us":399,"apply.run_wall_time_us":401,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":344,"prepare.run_wall_time_us":345,"replication_time_us":1184727,"thread_start_us":245,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":293,"prepare.run_cpu_time_us":339,"prepare.run_wall_time_us":339,"thread_start_us":191,"threads_started":1}]]}
I20240506 11:38:58.363981 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 35 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:58.363999 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 35 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:58.364701 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:58.365015 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:58.384929 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:58.388214 12574 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=33, attempt_no=0}) took 1228 ms. Trace:
I20240506 11:38:58.389192 12012 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 79) took 1078 ms. Trace:
I20240506 11:38:58.389214 12010 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 78) took 1078 ms. Trace:
I20240506 11:38:58.389336 12012 rpcz_store.cc:276] 0506 11:38:57.310408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:57.310467 (+ 59us) service_pool.cc:225] Handling call
0506 11:38:57.310691 (+ 224us) tablet_service.cc:2780] Created scanner 45b0e44cc2c14bde8a2955e87edc164d for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is d8c87f6ff5414f70a5878a34668404c5
0506 11:38:57.311677 (+ 986us) tablet_service.cc:2920] Creating iterator
0506 11:38:57.311724 (+ 47us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:57.454243 (+142519us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:58.387217 (+932974us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1075484 microseconds
0506 11:38:58.387270 (+ 53us) tablet_service.cc:2945] Iterator created
0506 11:38:58.387944 (+ 674us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:58.387977 (+ 33us) tablet_service.cc:3010] has_more: true
0506 11:38:58.388013 (+ 36us) tablet_service.cc:3027] Continuing scan request
0506 11:38:58.388096 (+ 83us) tablet_service.cc:3091] Found scanner 45b0e44cc2c14bde8a2955e87edc164d for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is d8c87f6ff5414f70a5878a34668404c5
0506 11:38:58.389114 (+ 1018us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":75776}
I20240506 11:38:58.388546 12574 rpcz_store.cc:276] 0506 11:38:57.158892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:57.159106 (+ 214us) service_pool.cc:225] Handling call
0506 11:38:58.387678 (+1228572us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:57.159739 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:57.159818 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:57.159821 (+ 3us) write_op.cc:435] Acquired schema lock
0506 11:38:57.159822 (+ 1us) tablet.cc:660] Decoding operations
0506 11:38:57.159905 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:57.159912 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:57.159915 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:57.159954 (+ 39us) tablet.cc:699] Row locks acquired
0506 11:38:57.159955 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:57.159981 (+ 26us) write_op.cc:270] Start()
0506 11:38:57.454111 (+294130us) write_op.cc:276] Timestamp: P: 1714995537159920 usec, L: 0
0506 11:38:57.454115 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:38:57.454396 (+ 281us) log.cc:842] Serialized 354 byte log entry
0506 11:38:58.386201 (+931805us) op_driver.cc:444] REPLICATION: finished
0506 11:38:58.386683 (+ 482us) write_op.cc:301] APPLY: starting
0506 11:38:58.386703 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:58.386755 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:58.386782 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:58.386869 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:58.386892 (+ 23us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:58.386897 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:38:58.386992 (+ 95us) log.cc:842] Serialized 22 byte log entry
0506 11:38:58.387560 (+ 568us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:58.387578 (+ 18us) write_op.cc:454] Released schema lock
0506 11:38:58.387585 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":399,"num_ops":1,"prepare.queue_time_us":262,"prepare.run_cpu_time_us":663,"prepare.run_wall_time_us":294724,"replication_time_us":932040,"thread_start_us":497,"threads_started":2}]]}
I20240506 11:38:58.389444 12010 rpcz_store.cc:276] 0506 11:38:57.310307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:57.310390 (+ 83us) service_pool.cc:225] Handling call
0506 11:38:57.310669 (+ 279us) tablet_service.cc:2780] Created scanner b3411d742d5a4d78bfaa7c7f8d982c4a for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is b646f5283c8e48c6af4453064c8bf9fc
0506 11:38:57.311686 (+ 1017us) tablet_service.cc:2920] Creating iterator
0506 11:38:57.311728 (+ 42us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:57.454270 (+142542us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:38:58.387125 (+932855us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1075383 microseconds
0506 11:38:58.387217 (+ 92us) tablet_service.cc:2945] Iterator created
0506 11:38:58.387952 (+ 735us) tablet_service.cc:2967] Iterator init: OK
0506 11:38:58.387983 (+ 31us) tablet_service.cc:3010] has_more: true
0506 11:38:58.388013 (+ 30us) tablet_service.cc:3027] Continuing scan request
0506 11:38:58.388096 (+ 83us) tablet_service.cc:3091] Found scanner b3411d742d5a4d78bfaa7c7f8d982c4a for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is b646f5283c8e48c6af4453064c8bf9fc
0506 11:38:58.389114 (+ 1018us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":69504}
W20240506 11:38:58.390426 12403 scanner-internal.cc:458] Time spent opening tablet: real 1.081s user 0.000s sys 0.000s
W20240506 11:38:58.390612 12404 scanner-internal.cc:458] Time spent opening tablet: real 1.081s user 0.000s sys 0.001s
I20240506 11:38:58.672322 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:58.672516 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:58.673408 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:58.673895 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 38 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:58.674306 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 38 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:58.674775 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:58.675356 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:38:59.241278 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:59.241472 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:59.242188 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:59.243080 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 37 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:38:59.243101 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 37 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:59.243664 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:38:59.407236 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:38:59.407457 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:38:59.408164 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:38:59.408682 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 39 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:38:59.408711 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 39 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:38:59.409511 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:38:59.410694 12581 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=35, attempt_no=0}) took 1019 ms. Trace:
I20240506 11:38:59.410835 12581 rpcz_store.cc:276] 0506 11:38:58.390825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:58.390867 (+ 42us) service_pool.cc:225] Handling call
0506 11:38:59.410635 (+1019768us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:58.480696 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:58.480816 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:58.480824 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:38:58.480827 (+ 3us) tablet.cc:660] Decoding operations
0506 11:38:58.480916 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:58.480928 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:58.480934 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:58.480986 (+ 52us) tablet.cc:699] Row locks acquired
0506 11:38:58.480987 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:38:58.481024 (+ 37us) write_op.cc:270] Start()
0506 11:38:58.910155 (+429131us) write_op.cc:276] Timestamp: P: 1714995538480967 usec, L: 0
0506 11:38:58.910160 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:38:58.910469 (+ 309us) log.cc:842] Serialized 354 byte log entry
0506 11:38:59.408680 (+498211us) op_driver.cc:444] REPLICATION: finished
0506 11:38:59.409465 (+ 785us) write_op.cc:301] APPLY: starting
0506 11:38:59.409486 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0506 11:38:59.409552 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
0506 11:38:59.409589 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:38:59.409667 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:38:59.409700 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:38:59.409707 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:38:59.409787 (+ 80us) log.cc:842] Serialized 22 byte log entry
0506 11:38:59.410477 (+ 690us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:38:59.410498 (+ 21us) write_op.cc:454] Released schema lock
0506 11:38:59.410505 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":691,"num_ops":1,"prepare.queue_time_us":89389,"prepare.run_cpu_time_us":800,"prepare.run_wall_time_us":429864,"replication_time_us":497989,"thread_start_us":600,"threads_started":1}]]}
W20240506 11:38:59.412384 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.871s user 0.001s sys 0.000s
W20240506 11:38:59.412659 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.872s user 0.000s sys 0.001s
I20240506 11:38:59.439606 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:00.016520 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:00.016721 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:00.017557 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:00.017941 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 41 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:00.017959 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 41 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:00.018633 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:00.018898 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:00.221904 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:01.016804 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:01.017030 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:01.017762 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:01.018236 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:01.018272 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:01.019006 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:01.019433 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:01.019904 12592 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=38, attempt_no=0}) took 1577 ms. Trace:
I20240506 11:39:01.020128 12592 rpcz_store.cc:276] 0506 11:38:59.442410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:59.442492 (+ 82us) service_pool.cc:225] Handling call
0506 11:39:01.019820 (+1577328us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:38:59.544126 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:38:59.544275 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:38:59.544281 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:38:59.544285 (+ 4us) tablet.cc:660] Decoding operations
0506 11:38:59.544380 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:38:59.544392 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:38:59.544396 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:38:59.544449 (+ 53us) tablet.cc:699] Row locks acquired
0506 11:38:59.544451 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:38:59.544497 (+ 46us) write_op.cc:270] Start()
0506 11:38:59.568624 (+ 24127us) write_op.cc:276] Timestamp: P: 1714995539544449 usec, L: 0
0506 11:38:59.568628 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:38:59.568835 (+ 207us) log.cc:842] Serialized 354 byte log entry
0506 11:39:01.018669 (+1449834us) op_driver.cc:444] REPLICATION: finished
0506 11:39:01.019253 (+ 584us) write_op.cc:301] APPLY: starting
0506 11:39:01.019272 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:01.019314 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:01.019349 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:01.019443 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:01.019475 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:01.019486 (+ 11us) write_op.cc:312] APPLY: finished
0506 11:39:01.019571 (+ 85us) log.cc:842] Serialized 22 byte log entry
0506 11:39:01.019685 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:01.019705 (+ 20us) write_op.cc:454] Released schema lock
0506 11:39:01.019713 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":507,"num_ops":1,"prepare.queue_time_us":101220,"prepare.run_cpu_time_us":702,"prepare.run_wall_time_us":24768,"replication_time_us":1449995,"thread_start_us":392,"threads_started":1}]]}
I20240506 11:39:01.021973 12012 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 87) took 1458 ms. Trace:
I20240506 11:39:01.022081 12012 rpcz_store.cc:276] 0506 11:38:59.563696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:59.563774 (+ 78us) service_pool.cc:225] Handling call
0506 11:38:59.564051 (+ 277us) tablet_service.cc:2780] Created scanner f888bd2c180f4ac99a4c710e59d3dc66 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 404bca1c049643bcbc688a2e9f0d65f2
0506 11:38:59.565036 (+ 985us) tablet_service.cc:2920] Creating iterator
0506 11:38:59.565076 (+ 40us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:59.568758 (+ 3682us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:01.019827 (+1451069us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1454740 microseconds
0506 11:39:01.019945 (+ 118us) tablet_service.cc:2945] Iterator created
0506 11:39:01.020641 (+ 696us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:01.020674 (+ 33us) tablet_service.cc:3010] has_more: true
0506 11:39:01.020713 (+ 39us) tablet_service.cc:3027] Continuing scan request
0506 11:39:01.020802 (+ 89us) tablet_service.cc:3091] Found scanner f888bd2c180f4ac99a4c710e59d3dc66 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 404bca1c049643bcbc688a2e9f0d65f2
0506 11:39:01.021903 (+ 1101us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":81664}
I20240506 11:39:01.022418 12010 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 88) took 1458 ms. Trace:
I20240506 11:39:01.022518 12010 rpcz_store.cc:276] 0506 11:38:59.563815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:38:59.563869 (+ 54us) service_pool.cc:225] Handling call
0506 11:38:59.564051 (+ 182us) tablet_service.cc:2780] Created scanner 66973e7a02fc40d89675f003faa65237 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 68479fe31684406b87d4c9f84cfc08b1
0506 11:38:59.565035 (+ 984us) tablet_service.cc:2920] Creating iterator
0506 11:38:59.565076 (+ 41us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:38:59.568758 (+ 3682us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:01.019732 (+1450974us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1454638 microseconds
0506 11:39:01.019848 (+ 116us) tablet_service.cc:2945] Iterator created
0506 11:39:01.021339 (+ 1491us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:01.021367 (+ 28us) tablet_service.cc:3010] has_more: true
0506 11:39:01.021404 (+ 37us) tablet_service.cc:3027] Continuing scan request
0506 11:39:01.021471 (+ 67us) tablet_service.cc:3091] Found scanner 66973e7a02fc40d89675f003faa65237 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 68479fe31684406b87d4c9f84cfc08b1
0506 11:39:01.022318 (+ 847us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":40960}
W20240506 11:39:01.022794 12403 scanner-internal.cc:458] Time spent opening tablet: real 1.460s user 0.000s sys 0.000s
W20240506 11:39:01.023118 12404 scanner-internal.cc:458] Time spent opening tablet: real 1.460s user 0.001s sys 0.000s
I20240506 11:39:01.224400 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:01.224640 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:01.225426 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:01.225948 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:01.225981 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 43 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:01.226050 12592 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=39, attempt_no=0}) took 1205 ms. Trace:
I20240506 11:39:01.226138 12592 rpcz_store.cc:276] 0506 11:39:00.020452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:00.020535 (+ 83us) service_pool.cc:225] Handling call
0506 11:39:01.225985 (+1205450us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:00.020941 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:00.021035 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:00.021039 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:00.021040 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:00.021105 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:00.021112 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:00.021115 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:00.021149 (+ 34us) tablet.cc:699] Row locks acquired
0506 11:39:00.021150 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:00.021181 (+ 31us) write_op.cc:270] Start()
0506 11:39:00.989355 (+968174us) write_op.cc:276] Timestamp: P: 1714995540021141 usec, L: 0
0506 11:39:00.989360 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:00.989636 (+ 276us) log.cc:842] Serialized 354 byte log entry
0506 11:39:01.225314 (+235678us) op_driver.cc:444] REPLICATION: finished
0506 11:39:01.225437 (+ 123us) write_op.cc:301] APPLY: starting
0506 11:39:01.225455 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:01.225502 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:01.225541 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:01.225614 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:01.225655 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:01.225679 (+ 24us) write_op.cc:312] APPLY: finished
0506 11:39:01.225763 (+ 84us) log.cc:842] Serialized 22 byte log entry
0506 11:39:01.225839 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:01.225856 (+ 17us) write_op.cc:454] Released schema lock
0506 11:39:01.225863 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":65,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":691,"prepare.run_wall_time_us":968773,"replication_time_us":235925}]]}
I20240506 11:39:01.226727 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:01.905625 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:01.905683 12182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:43381 (request call id 2147) took 1682 ms. Trace:
I20240506 11:39:01.905974 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:01.906829 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:01.907310 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:01.906438 12182 rpcz_store.cc:276] 0506 11:39:00.222997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:00.223052 (+ 55us) service_pool.cc:225] Handling call
0506 11:39:00.223231 (+ 179us) raft_consensus.cc:1457] Updating replica for 3 ops
0506 11:39:00.223368 (+ 137us) raft_consensus.cc:1520] Early marking committed up to index 39
0506 11:39:00.223451 (+ 83us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0506 11:39:00.223538 (+ 87us) write_op.cc:270] Start()
0506 11:39:00.286697 (+ 63159us) write_op.cc:276] Timestamp: P: 1714995538910957 usec, L: 0
0506 11:39:00.287026 (+ 329us) write_op.cc:270] Start()
0506 11:39:01.088177 (+801151us) write_op.cc:276] Timestamp: P: 1714995539413601 usec, L: 0
0506 11:39:01.088757 (+ 580us) write_op.cc:270] Start()
0506 11:39:01.903918 (+815161us) write_op.cc:276] Timestamp: P: 1714995539544449 usec, L: 0
0506 11:39:01.904576 (+ 658us) log.cc:842] Serialized 1062 byte log entry
0506 11:39:01.904903 (+ 327us) raft_consensus.cc:1631] Marking committed up to 41
0506 11:39:01.905501 (+ 598us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:01.905534 (+ 33us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:01.905548 (+ 14us) raft_consensus.cc:1671] finished
0506 11:39:01.905550 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:01.905642 (+ 92us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:00.287004 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:00.287123 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:00.287128 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:00.287130 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:00.287203 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:00.287215 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:00.287218 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:00.287279 (+ 61us) tablet.cc:699] Row locks acquired
0506 11:39:00.287281 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:01.904947 (+1617666us) op_driver.cc:444] REPLICATION: finished
0506 11:39:01.905337 (+ 390us) write_op.cc:301] APPLY: starting
0506 11:39:01.905355 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:01.905404 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:01.905408 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:01.905524 (+ 116us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:01.905559 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:01.905610 (+ 51us) write_op.cc:312] APPLY: finished
0506 11:39:01.905706 (+ 96us) log.cc:842] Serialized 22 byte log entry
0506 11:39:01.905785 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:01.905805 (+ 20us) write_op.cc:454] Released schema lock
0506 11:39:01.905810 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0506 11:39:01.905928 (+ 118us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:01.905933 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:39:01.088677 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:01.088778 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:01.088784 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:01.088785 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:01.088860 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:01.088870 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:01.088873 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:01.088919 (+ 46us) tablet.cc:699] Row locks acquired
0506 11:39:01.088920 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:01.905232 (+816312us) op_driver.cc:444] REPLICATION: finished
0506 11:39:01.905628 (+ 396us) write_op.cc:301] APPLY: starting
0506 11:39:01.905642 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:01.905670 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:01.905672 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:01.905733 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:01.905755 (+ 22us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:01.905817 (+ 62us) write_op.cc:312] APPLY: finished
0506 11:39:01.905959 (+ 142us) log.cc:842] Serialized 22 byte log entry
0506 11:39:01.906051 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:01.906065 (+ 14us) write_op.cc:454] Released schema lock
0506 11:39:01.906069 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0506 11:39:01.906126 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:01.906129 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:39:01.904438 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:01.904560 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:01.904565 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:01.904567 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:01.904641 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:01.904654 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:01.904657 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:01.904713 (+ 56us) tablet.cc:699] Row locks acquired
0506 11:39:01.904715 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":248,"threads_started":1,"wal-append.queue_time_us":338,"child_traces":[["op",{"apply.queue_time_us":319,"apply.run_cpu_time_us":672,"apply.run_wall_time_us":672,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":409,"prepare.run_wall_time_us":426,"replication_time_us":1681392,"thread_start_us":209,"threads_started":1}],["op",{"apply.queue_time_us":308,"apply.run_cpu_time_us":524,"apply.run_wall_time_us":538,"mutex_wait_us":24,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":367,"prepare.run_cpu_time_us":323,"prepare.run_wall_time_us":324,"replication_time_us":1618192,"thread_start_us":461,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":373,"prepare.run_cpu_time_us":351,"prepare.run_wall_time_us":351,"thread_start_us":236,"threads_started":1}]]}
I20240506 11:39:01.907423 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:01.908463 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:01.908753 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:02.148749 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:02.151252 12602 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=40, attempt_no=0}) took 1129 ms. Trace:
I20240506 11:39:02.151522 12602 rpcz_store.cc:276] 0506 11:39:01.022198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:01.022266 (+ 68us) service_pool.cc:225] Handling call
0506 11:39:02.151181 (+1128915us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:01.023050 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:01.023124 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:01.023128 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:01.023130 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:01.023189 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:01.023198 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:01.023201 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:01.023235 (+ 34us) tablet.cc:699] Row locks acquired
0506 11:39:01.023237 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:01.023272 (+ 35us) write_op.cc:270] Start()
0506 11:39:01.026410 (+ 3138us) write_op.cc:276] Timestamp: P: 1714995541023225 usec, L: 0
0506 11:39:01.026413 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 11:39:01.026573 (+ 160us) log.cc:842] Serialized 354 byte log entry
0506 11:39:02.150075 (+1123502us) op_driver.cc:444] REPLICATION: finished
0506 11:39:02.150566 (+ 491us) write_op.cc:301] APPLY: starting
0506 11:39:02.150582 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:02.150615 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:02.150660 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:02.150756 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:02.150790 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:02.150796 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:39:02.150900 (+ 104us) log.cc:842] Serialized 22 byte log entry
0506 11:39:02.151049 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:02.151071 (+ 22us) write_op.cc:454] Released schema lock
0506 11:39:02.151080 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":434,"num_ops":1,"prepare.queue_time_us":228,"prepare.run_cpu_time_us":504,"prepare.run_wall_time_us":3584,"replication_time_us":1123628,"thread_start_us":244,"threads_started":1}]]}
W20240506 11:39:02.153806 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.980s user 0.001s sys 0.000s
W20240506 11:39:02.153762 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.980s user 0.001s sys 0.000s
I20240506 11:39:02.941139 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:02.941395 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:02.942092 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:02.942629 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:02.942757 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:02.943387 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:02.944089 12609 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=41, attempt_no=0}) took 1715 ms. Trace:
I20240506 11:39:02.944193 12609 rpcz_store.cc:276] 0506 11:39:01.228167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:01.228231 (+ 64us) service_pool.cc:225] Handling call
0506 11:39:02.944030 (+1715799us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:01.228685 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:01.228763 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:01.228767 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:01.228768 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:01.228828 (+ 60us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:01.228837 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:01.228839 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:01.228872 (+ 33us) tablet.cc:699] Row locks acquired
0506 11:39:01.228874 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:01.228903 (+ 29us) write_op.cc:270] Start()
0506 11:39:01.241024 (+ 12121us) write_op.cc:276] Timestamp: P: 1714995541228844 usec, L: 0
0506 11:39:01.241028 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:01.241205 (+ 177us) log.cc:842] Serialized 354 byte log entry
0506 11:39:02.942744 (+1701539us) op_driver.cc:444] REPLICATION: finished
0506 11:39:02.943495 (+ 751us) write_op.cc:301] APPLY: starting
0506 11:39:02.943514 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:02.943558 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:02.943647 (+ 89us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:02.943722 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:02.943759 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:02.943766 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:39:02.943837 (+ 71us) log.cc:842] Serialized 22 byte log entry
0506 11:39:02.943904 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:02.943922 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:02.943930 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":669,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":533,"prepare.run_wall_time_us":12596,"replication_time_us":1701685,"thread_start_us":1269,"threads_started":1}]]}
I20240506 11:39:03.003793 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 11:39:03.007553 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.703s user 0.001s sys 0.000s
W20240506 11:39:03.007722 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.703s user 0.000s sys 0.000s
I20240506 11:39:03.513851 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:03.514052 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:03.514155 12182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:43381 (request call id 2148) took 1604 ms. Trace:
I20240506 11:39:03.514771 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:03.514292 12182 rpcz_store.cc:276] 0506 11:39:01.909360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:01.909438 (+ 78us) service_pool.cc:225] Handling call
0506 11:39:01.909634 (+ 196us) raft_consensus.cc:1457] Updating replica for 3 ops
0506 11:39:01.909790 (+ 156us) raft_consensus.cc:1520] Early marking committed up to index 42
0506 11:39:01.909867 (+ 77us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0506 11:39:01.910141 (+ 274us) write_op.cc:270] Start()
0506 11:39:02.450303 (+540162us) write_op.cc:276] Timestamp: P: 1714995540021141 usec, L: 0
0506 11:39:02.450970 (+ 667us) write_op.cc:270] Start()
0506 11:39:03.163101 (+712131us) write_op.cc:276] Timestamp: P: 1714995541023225 usec, L: 0
0506 11:39:03.163569 (+ 468us) write_op.cc:270] Start()
0506 11:39:03.512731 (+349162us) write_op.cc:276] Timestamp: P: 1714995541228844 usec, L: 0
0506 11:39:03.513074 (+ 343us) log.cc:842] Serialized 1062 byte log entry
0506 11:39:03.513410 (+ 336us) raft_consensus.cc:1631] Marking committed up to 43
0506 11:39:03.513740 (+ 330us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:03.513777 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:03.513984 (+ 207us) raft_consensus.cc:1671] finished
0506 11:39:03.513986 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:03.514114 (+ 128us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:02.450844 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:02.451026 (+ 182us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:02.451031 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:02.451032 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:02.451110 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:02.451125 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:02.451128 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:02.451181 (+ 53us) tablet.cc:699] Row locks acquired
0506 11:39:02.451182 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:03.513487 (+1062305us) op_driver.cc:444] REPLICATION: finished
0506 11:39:03.513832 (+ 345us) write_op.cc:301] APPLY: starting
0506 11:39:03.513849 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:03.513889 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:03.513893 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:03.513961 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:03.513992 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:03.514051 (+ 59us) write_op.cc:312] APPLY: finished
0506 11:39:03.514143 (+ 92us) log.cc:842] Serialized 22 byte log entry
0506 11:39:03.514210 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:03.514226 (+ 16us) write_op.cc:454] Released schema lock
0506 11:39:03.514233 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 11:39:03.514305 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:03.514308 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:39:03.163549 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:03.163634 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:03.163639 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:03.163640 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:03.163710 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:03.163720 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:03.163723 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:03.163765 (+ 42us) tablet.cc:699] Row locks acquired
0506 11:39:03.163766 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0506 11:39:03.512938 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:03.513040 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:03.513044 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:03.513046 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:03.513119 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:03.513130 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:03.513133 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:03.513182 (+ 49us) tablet.cc:699] Row locks acquired
0506 11:39:03.513184 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":260,"threads_started":1,"wal-append.queue_time_us":338,"child_traces":[["op",{"apply.queue_time_us":283,"apply.run_cpu_time_us":540,"apply.run_wall_time_us":540,"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":388,"prepare.run_cpu_time_us":427,"prepare.run_wall_time_us":427,"replication_time_us":1603294,"thread_start_us":440,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":327,"prepare.run_cpu_time_us":288,"prepare.run_wall_time_us":288,"thread_start_us":190,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":309,"prepare.run_wall_time_us":309}]]}
I20240506 11:39:03.515340 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:03.515383 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 45 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:03.516130 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:03.516503 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:04.051728 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:04.051923 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:04.053014 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 47 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:04.053556 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 47 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:04.054049 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:04.054117 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:04.055310 12619 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=43, attempt_no=0}) took 1109 ms. Trace:
I20240506 11:39:04.055425 12619 rpcz_store.cc:276] 0506 11:39:02.945699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:02.945767 (+ 68us) service_pool.cc:225] Handling call
0506 11:39:04.055260 (+1109493us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:02.946219 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:02.946303 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:02.946307 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:02.946309 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:02.946372 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:02.946380 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:02.946383 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:02.946417 (+ 34us) tablet.cc:699] Row locks acquired
0506 11:39:02.946418 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:02.946447 (+ 29us) write_op.cc:270] Start()
0506 11:39:03.128608 (+182161us) write_op.cc:276] Timestamp: P: 1714995542946422 usec, L: 0
0506 11:39:03.128612 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:03.128826 (+ 214us) log.cc:842] Serialized 354 byte log entry
0506 11:39:04.053619 (+924793us) op_driver.cc:444] REPLICATION: finished
0506 11:39:04.054718 (+ 1099us) write_op.cc:301] APPLY: starting
0506 11:39:04.054736 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:04.054777 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:04.054804 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:04.054904 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:04.054937 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:04.054964 (+ 27us) write_op.cc:312] APPLY: finished
0506 11:39:04.055059 (+ 95us) log.cc:842] Serialized 22 byte log entry
0506 11:39:04.055163 (+ 104us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:04.055176 (+ 13us) write_op.cc:454] Released schema lock
0506 11:39:04.055181 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1032,"num_ops":1,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":610,"prepare.run_wall_time_us":182681,"replication_time_us":924984,"thread_start_us":936,"threads_started":1}]]}
I20240506 11:39:04.679917 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:04.680096 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:04.680877 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:04.681368 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 48 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:04.681532 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 48 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:04.682134 12182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:43381 (request call id 2149) took 1165 ms. Trace:
I20240506 11:39:04.682206 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:04.682271 12182 rpcz_store.cc:276] 0506 11:39:03.516232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:03.516294 (+ 62us) service_pool.cc:225] Handling call
0506 11:39:03.516504 (+ 210us) raft_consensus.cc:1457] Updating replica for 3 ops
0506 11:39:03.516767 (+ 263us) raft_consensus.cc:1520] Early marking committed up to index 45
0506 11:39:03.517099 (+ 332us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0506 11:39:03.517207 (+ 108us) write_op.cc:270] Start()
0506 11:39:03.777351 (+260144us) write_op.cc:276] Timestamp: P: 1714995542155264 usec, L: 0
0506 11:39:03.777751 (+ 400us) write_op.cc:270] Start()
0506 11:39:03.905921 (+128170us) write_op.cc:276] Timestamp: P: 1714995542946422 usec, L: 0
0506 11:39:03.906272 (+ 351us) write_op.cc:270] Start()
0506 11:39:04.678452 (+772180us) write_op.cc:276] Timestamp: P: 1714995543129225 usec, L: 0
0506 11:39:04.679192 (+ 740us) log.cc:842] Serialized 1062 byte log entry
0506 11:39:04.679517 (+ 325us) raft_consensus.cc:1631] Marking committed up to 46
0506 11:39:04.679803 (+ 286us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:04.679844 (+ 41us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:04.682012 (+ 2168us) raft_consensus.cc:1671] finished
0506 11:39:04.682015 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:04.682088 (+ 73us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:03.777613 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:03.777741 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:03.777747 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:03.777749 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:03.777855 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:03.777872 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:03.777877 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:03.777932 (+ 55us) tablet.cc:699] Row locks acquired
0506 11:39:03.777934 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:04.679559 (+901625us) op_driver.cc:444] REPLICATION: finished
0506 11:39:04.679872 (+ 313us) write_op.cc:301] APPLY: starting
0506 11:39:04.679888 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:04.679927 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:04.679928 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:04.679985 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:04.680020 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:04.680046 (+ 26us) write_op.cc:312] APPLY: finished
0506 11:39:04.680124 (+ 78us) log.cc:842] Serialized 22 byte log entry
0506 11:39:04.680175 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:04.680191 (+ 16us) write_op.cc:454] Released schema lock
0506 11:39:04.680199 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0506 11:39:04.680264 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:04.680268 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:39:03.906125 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:03.906283 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:03.906290 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:39:03.906295 (+ 5us) tablet.cc:660] Decoding operations
0506 11:39:03.906381 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:03.906393 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:03.906397 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:03.906443 (+ 46us) tablet.cc:699] Row locks acquired
0506 11:39:03.906445 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0506 11:39:04.678993 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:04.679097 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:04.679102 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:04.679103 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:04.679182 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:04.679195 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:04.679198 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:04.679244 (+ 46us) tablet.cc:699] Row locks acquired
0506 11:39:04.679246 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":303872,"thread_start_us":227,"threads_started":1,"wal-append.queue_time_us":310,"child_traces":[["op",{"apply.queue_time_us":260,"apply.run_cpu_time_us":456,"apply.run_wall_time_us":455,"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":74,"prepare.run_cpu_time_us":420,"prepare.run_wall_time_us":420,"replication_time_us":1162338,"thread_start_us":186,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":407,"prepare.run_wall_time_us":407}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":386,"prepare.run_cpu_time_us":309,"prepare.run_wall_time_us":310,"thread_start_us":271,"threads_started":1}]]}
I20240506 11:39:04.683108 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:04.685892 12628 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=44, attempt_no=0}) took 1678 ms. Trace:
I20240506 11:39:04.686046 12628 rpcz_store.cc:276] 0506 11:39:03.007777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:03.007891 (+ 114us) service_pool.cc:225] Handling call
0506 11:39:04.685826 (+1677935us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:03.128934 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:03.129054 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:03.129060 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:03.129062 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:03.129151 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:03.129164 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:03.129171 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:03.129227 (+ 56us) tablet.cc:699] Row locks acquired
0506 11:39:03.129229 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:03.129271 (+ 42us) write_op.cc:270] Start()
0506 11:39:03.150411 (+ 21140us) write_op.cc:276] Timestamp: P: 1714995543129225 usec, L: 0
0506 11:39:03.150415 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:03.150607 (+ 192us) log.cc:842] Serialized 354 byte log entry
0506 11:39:04.684167 (+1533560us) op_driver.cc:444] REPLICATION: finished
0506 11:39:04.684830 (+ 663us) write_op.cc:301] APPLY: starting
0506 11:39:04.684845 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:04.684888 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:04.684908 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:04.684985 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:04.685031 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:04.685037 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:39:04.685133 (+ 96us) log.cc:842] Serialized 22 byte log entry
0506 11:39:04.685570 (+ 437us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:04.685734 (+ 164us) write_op.cc:454] Released schema lock
0506 11:39:04.685741 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":604,"num_ops":1,"prepare.queue_time_us":120624,"prepare.run_cpu_time_us":688,"prepare.run_wall_time_us":21755,"replication_time_us":1533725,"thread_start_us":497,"threads_started":1}]]}
I20240506 11:39:04.687340 12011 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 100) took 1528 ms. Trace:
I20240506 11:39:04.687451 12010 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 99) took 1528 ms. Trace:
I20240506 11:39:04.687440 12011 rpcz_store.cc:276] 0506 11:39:03.159016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:03.159090 (+ 74us) service_pool.cc:225] Handling call
0506 11:39:03.159272 (+ 182us) tablet_service.cc:2780] Created scanner b0458cb3fdc3402eb762ac268392304e for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 1f8b9f270c9943e0b7e6f2e7d1b7835c
0506 11:39:03.159944 (+ 672us) tablet_service.cc:2920] Creating iterator
0506 11:39:03.159965 (+ 21us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:03.159971 (+ 6us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:04.685628 (+1525657us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1525637 microseconds
0506 11:39:04.685726 (+ 98us) tablet_service.cc:2945] Iterator created
0506 11:39:04.686313 (+ 587us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:04.686342 (+ 29us) tablet_service.cc:3010] has_more: true
0506 11:39:04.686373 (+ 31us) tablet_service.cc:3027] Continuing scan request
0506 11:39:04.686461 (+ 88us) tablet_service.cc:3091] Found scanner b0458cb3fdc3402eb762ac268392304e for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 1f8b9f270c9943e0b7e6f2e7d1b7835c
0506 11:39:04.687296 (+ 835us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":7936}
I20240506 11:39:04.687573 12010 rpcz_store.cc:276] 0506 11:39:03.158814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:03.158888 (+ 74us) service_pool.cc:225] Handling call
0506 11:39:03.159139 (+ 251us) tablet_service.cc:2780] Created scanner 5e432853f8ed4e98bcd59df7071923e7 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 7bd66f40828540a6bbde285ae0e9550f
0506 11:39:03.159821 (+ 682us) tablet_service.cc:2920] Creating iterator
0506 11:39:03.159869 (+ 48us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:03.159877 (+ 8us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:04.685602 (+1525725us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1525717 microseconds
0506 11:39:04.685715 (+ 113us) tablet_service.cc:2945] Iterator created
0506 11:39:04.686274 (+ 559us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:04.686301 (+ 27us) tablet_service.cc:3010] has_more: true
0506 11:39:04.686332 (+ 31us) tablet_service.cc:3027] Continuing scan request
0506 11:39:04.686412 (+ 80us) tablet_service.cc:3091] Found scanner 5e432853f8ed4e98bcd59df7071923e7 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 7bd66f40828540a6bbde285ae0e9550f
0506 11:39:04.687389 (+ 977us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20240506 11:39:04.688235 12404 scanner-internal.cc:458] Time spent opening tablet: real 1.530s user 0.000s sys 0.000s
W20240506 11:39:04.688521 12403 scanner-internal.cc:458] Time spent opening tablet: real 1.530s user 0.001s sys 0.000s
I20240506 11:39:04.769752 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:04.835040 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:04.835227 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:04.835956 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:04.836428 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 49 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:04.836486 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 49 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:04.837364 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:04.837736 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:05.225260 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:05.225492 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:05.226171 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:05.226719 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:05.226728 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:05.227397 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:05.229071 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:05.237287 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:05.237457 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:05.238204 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:05.238548 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 49 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:05.238797 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 49 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:05.239346 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:06.159955 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:06.421514 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:06.421702 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:06.422367 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:06.422868 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 51 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:06.422889 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 51 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:06.423585 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:06.424224 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:06.425642 12638 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=47, attempt_no=0}) took 1587 ms. Trace:
I20240506 11:39:06.425761 12638 rpcz_store.cc:276] 0506 11:39:04.838207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:04.838259 (+ 52us) service_pool.cc:225] Handling call
0506 11:39:06.425592 (+1587333us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:05.185127 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:05.185218 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:05.185223 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:05.185224 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:05.185295 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:05.185304 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:05.185307 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:05.185348 (+ 41us) tablet.cc:699] Row locks acquired
0506 11:39:05.185350 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:05.185383 (+ 33us) write_op.cc:270] Start()
0506 11:39:05.696540 (+511157us) write_op.cc:276] Timestamp: P: 1714995545185331 usec, L: 0
0506 11:39:05.696544 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:05.696810 (+ 266us) log.cc:842] Serialized 354 byte log entry
0506 11:39:06.422876 (+726066us) op_driver.cc:444] REPLICATION: finished
0506 11:39:06.423779 (+ 903us) write_op.cc:301] APPLY: starting
0506 11:39:06.423799 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:06.423846 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:06.423870 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:06.423979 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:06.424026 (+ 47us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:06.424038 (+ 12us) write_op.cc:312] APPLY: finished
0506 11:39:06.424162 (+ 124us) log.cc:842] Serialized 22 byte log entry
0506 11:39:06.425471 (+ 1309us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:06.425485 (+ 14us) write_op.cc:454] Released schema lock
0506 11:39:06.425490 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":819,"num_ops":1,"prepare.queue_time_us":346432,"prepare.run_cpu_time_us":676,"prepare.run_wall_time_us":511754,"replication_time_us":726304,"thread_start_us":709,"threads_started":1}]]}
I20240506 11:39:06.426580 12010 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 107) took 1046 ms. Trace:
I20240506 11:39:06.426682 12010 rpcz_store.cc:276] 0506 11:39:05.380427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:05.380495 (+ 68us) service_pool.cc:225] Handling call
0506 11:39:05.380708 (+ 213us) tablet_service.cc:2780] Created scanner 36727be381394bfcbd8a8aa8b736042e for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 1c86923ada9447879a0f191b26d1b6f6
0506 11:39:05.381512 (+ 804us) tablet_service.cc:2920] Creating iterator
0506 11:39:05.381547 (+ 35us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:05.696654 (+315107us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:06.424647 (+727993us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1043082 microseconds
0506 11:39:06.424825 (+ 178us) tablet_service.cc:2945] Iterator created
0506 11:39:06.425294 (+ 469us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:06.425321 (+ 27us) tablet_service.cc:3010] has_more: true
0506 11:39:06.425352 (+ 31us) tablet_service.cc:3027] Continuing scan request
0506 11:39:06.425441 (+ 89us) tablet_service.cc:3091] Found scanner 36727be381394bfcbd8a8aa8b736042e for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 1c86923ada9447879a0f191b26d1b6f6
0506 11:39:06.426494 (+ 1053us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":7424}
I20240506 11:39:06.426872 12011 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50464 (request call id 108) took 1044 ms. Trace:
I20240506 11:39:06.426988 12011 rpcz_store.cc:276] 0506 11:39:05.381773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:05.381834 (+ 61us) service_pool.cc:225] Handling call
0506 11:39:05.382019 (+ 185us) tablet_service.cc:2780] Created scanner dcdd3c93b1d5438d9f349af26660fff9 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 48eb6b451c0e4b6b96e23b84ddfe1dc4
0506 11:39:05.382715 (+ 696us) tablet_service.cc:2920] Creating iterator
0506 11:39:05.382746 (+ 31us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:05.696694 (+313948us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:06.424646 (+727952us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1041883 microseconds
0506 11:39:06.424784 (+ 138us) tablet_service.cc:2945] Iterator created
0506 11:39:06.425309 (+ 525us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:06.425352 (+ 43us) tablet_service.cc:3010] has_more: true
0506 11:39:06.425381 (+ 29us) tablet_service.cc:3027] Continuing scan request
0506 11:39:06.425450 (+ 69us) tablet_service.cc:3091] Found scanner dcdd3c93b1d5438d9f349af26660fff9 for tablet 63d657e07d5047b98c616f9a175d5cfe, query id is 48eb6b451c0e4b6b96e23b84ddfe1dc4
0506 11:39:06.426506 (+ 1056us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":17664}
W20240506 11:39:06.427955 12404 scanner-internal.cc:458] Time spent opening tablet: real 1.048s user 0.001s sys 0.000s
W20240506 11:39:06.427955 12403 scanner-internal.cc:458] Time spent opening tablet: real 1.048s user 0.001s sys 0.000s
I20240506 11:39:06.931669 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:06.931850 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:06.932724 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:06.933231 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 51 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:06.933264 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 51 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:06.933863 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:06.934131 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:07.088966 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:07.089179 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:07.089960 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:07.090528 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 52 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:07.090554 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 52 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:07.091234 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:07.091312 12644 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=48, attempt_no=0}) took 1862 ms. Trace:
I20240506 11:39:07.091413 12644 rpcz_store.cc:276] 0506 11:39:05.229310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:05.229399 (+ 89us) service_pool.cc:225] Handling call
0506 11:39:07.091252 (+1861853us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:05.696918 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:05.697062 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:05.697068 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:05.697070 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:05.697194 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:05.697208 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:05.697216 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:05.697274 (+ 58us) tablet.cc:699] Row locks acquired
0506 11:39:05.697277 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:39:05.697323 (+ 46us) write_op.cc:270] Start()
0506 11:39:06.647465 (+950142us) write_op.cc:276] Timestamp: P: 1714995545697271 usec, L: 0
0506 11:39:06.647473 (+ 8us) op_driver.cc:331] REPLICATION: starting
0506 11:39:06.647777 (+ 304us) log.cc:842] Serialized 354 byte log entry
0506 11:39:07.090004 (+442227us) op_driver.cc:444] REPLICATION: finished
0506 11:39:07.090674 (+ 670us) write_op.cc:301] APPLY: starting
0506 11:39:07.090693 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:07.090759 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:07.090788 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:07.090863 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:07.090901 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:07.090909 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:39:07.091014 (+ 105us) log.cc:842] Serialized 22 byte log entry
0506 11:39:07.091116 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:07.091137 (+ 21us) write_op.cc:454] Released schema lock
0506 11:39:07.091145 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":612,"num_ops":1,"prepare.queue_time_us":467026,"prepare.run_cpu_time_us":851,"prepare.run_wall_time_us":950920,"replication_time_us":442498,"thread_start_us":525,"threads_started":1}]]}
I20240506 11:39:07.093645 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 11:39:07.094069 12403 scanner-internal.cc:458] Time spent opening tablet: real 0.515s user 0.001s sys 0.000s
W20240506 11:39:07.094219 12404 scanner-internal.cc:458] Time spent opening tablet: real 0.515s user 0.001s sys 0.000s
I20240506 11:39:07.263733 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:07.263926 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:07.264709 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:07.265583 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 52 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:07.266302 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 52 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:07.266918 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:07.267333 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:07.621601 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:07.621803 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:07.622785 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:07.623418 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 53 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:07.625382 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 53 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:07.626034 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:07.626430 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:07.626616 12648 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=49, attempt_no=0}) took 1198 ms. Trace:
I20240506 11:39:07.626725 12648 rpcz_store.cc:276] 0506 11:39:06.427936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:06.427995 (+ 59us) service_pool.cc:225] Handling call
0506 11:39:07.626544 (+1198549us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:06.647896 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:06.648017 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:06.648032 (+ 15us) write_op.cc:435] Acquired schema lock
0506 11:39:06.648034 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:06.648133 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:06.648144 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:06.648148 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:06.648203 (+ 55us) tablet.cc:699] Row locks acquired
0506 11:39:06.648206 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:39:06.648245 (+ 39us) write_op.cc:270] Start()
0506 11:39:07.582432 (+934187us) write_op.cc:276] Timestamp: P: 1714995546648211 usec, L: 0
0506 11:39:07.582437 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:07.582737 (+ 300us) log.cc:842] Serialized 354 byte log entry
0506 11:39:07.624503 (+ 41766us) op_driver.cc:444] REPLICATION: finished
0506 11:39:07.625881 (+ 1378us) write_op.cc:301] APPLY: starting
0506 11:39:07.625899 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:07.625936 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:07.625982 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:07.626072 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:07.626102 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:07.626109 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:39:07.626195 (+ 86us) log.cc:842] Serialized 22 byte log entry
0506 11:39:07.626303 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:07.626443 (+ 140us) write_op.cc:454] Released schema lock
0506 11:39:07.626449 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1315,"num_ops":1,"prepare.queue_time_us":219447,"prepare.run_cpu_time_us":853,"prepare.run_wall_time_us":934965,"raft.queue_time_us":39,"raft.run_cpu_time_us":40,"raft.run_wall_time_us":41,"replication_time_us":41878,"spinlock_wait_cycles":104704,"thread_start_us":525,"threads_started":1}]]}
I20240506 11:39:08.512177 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:08.512363 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:08.513316 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:08.513752 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 53 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:08.513767 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 53 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:08.514349 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:08.678679 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:08.680884 12655 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50464 (ReqId={client: 356934217ee54e7e981e45a1a08f241f, seq_no=50, attempt_no=0}) took 1586 ms. Trace:
I20240506 11:39:08.681017 12655 rpcz_store.cc:276] 0506 11:39:07.094044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:07.094121 (+ 77us) service_pool.cc:225] Handling call
0506 11:39:08.680863 (+1586742us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:07.582894 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63d657e07d5047b98c616f9a175d5cfe
0506 11:39:07.583089 (+ 195us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:07.583095 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:07.583098 (+ 3us) tablet.cc:660] Decoding operations
0506 11:39:07.583183 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:07.583193 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:07.583196 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:07.583240 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:39:07.583241 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:07.583278 (+ 37us) write_op.cc:270] Start()
0506 11:39:07.959466 (+376188us) write_op.cc:276] Timestamp: P: 1714995547583225 usec, L: 0
0506 11:39:07.959472 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:39:07.959726 (+ 254us) log.cc:842] Serialized 354 byte log entry
0506 11:39:08.679992 (+720266us) op_driver.cc:444] REPLICATION: finished
0506 11:39:08.680376 (+ 384us) write_op.cc:301] APPLY: starting
0506 11:39:08.680396 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:08.680462 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:08.680489 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:08.680573 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:08.680600 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:08.680605 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:08.680687 (+ 82us) log.cc:842] Serialized 22 byte log entry
0506 11:39:08.680753 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:08.680770 (+ 17us) write_op.cc:454] Released schema lock
0506 11:39:08.680775 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":324,"num_ops":1,"prepare.queue_time_us":488339,"prepare.run_cpu_time_us":798,"prepare.run_wall_time_us":376904,"replication_time_us":720497,"thread_start_us":226,"threads_started":1}]]}
I20240506 11:39:08.805585 12032 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240506 11:39:08.814699 12293 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:39:08.815919 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:08.816088 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:08.817034 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:08.818356 12313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:08.819988 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:08.820674 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:08.822120 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:08.824297 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:08.824487 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:08.825110 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:08.825832 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:08.826208 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:08.826875 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:08.827162 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240506 11:39:08.829780 12163 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:39:08.839213 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:08.839371 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:08.839988 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:08.840526 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:08.841053 12182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:08.841899 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:08.842312 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e5939149a922434d86d9213ce9d3501d | 127.10.45.190:37311 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:40309 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
b6a0d74bcf824aebbbb86cf0fd12f16e | 127.10.45.129:39317 | HEALTHY | <none> | 1 | 0
e36d0b9df4204b1d8a524fd2a2f78042 | 127.10.45.131:34507 | HEALTHY | <none> | 0 | 0
f1dfb32eb83944708aa5de1a681ed391 | 127.10.45.130:45689 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.45.129 | experimental | 127.10.45.129:39317
local_ip_for_outbound_sockets | 127.10.45.130 | experimental | 127.10.45.130:45689
local_ip_for_outbound_sockets | 127.10.45.131 | experimental | 127.10.45.131:34507
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.45.129:39317
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.45.130:45689
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.45.131:34507
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:40309 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240506 11:39:08.878983 10422 log_verifier.cc:126] Checking tablet 63d657e07d5047b98c616f9a175d5cfe
I20240506 11:39:08.934401 10422 log_verifier.cc:177] Verified matching terms for 54 ops in tablet 63d657e07d5047b98c616f9a175d5cfe
I20240506 11:39:08.963021 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 11970
I20240506 11:39:08.970356 12092 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:39:08.977299 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:08.977469 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:08.977905 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:08.978058 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:08.978174 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:08.978541 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:08.978766 12312 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1dfb32eb83944708aa5de1a681ed391 in term 5.
I20240506 11:39:08.978801 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:08.979169 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:08.979272 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:39:08.979370 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 5.
I20240506 11:39:08.979497 12475 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 11:39:08.979611 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:08.979707 12475 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 5 FOLLOWER]: Advancing to term 6
I20240506 11:39:08.979737 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:39:08.979984 12541 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240506 11:39:08.980091 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b6a0d74bcf824aebbbb86cf0fd12f16e)
I20240506 11:39:08.980171 12541 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 5 FOLLOWER]: Advancing to term 6
I20240506 11:39:08.981845 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:08.982729 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042"
I20240506 11:39:08.982816 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 election: Requested vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:08.982986 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:08.983352 12312 raft_consensus.cc:2379] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 in current term 6: Already voted for candidate e36d0b9df4204b1d8a524fd2a2f78042 in this term.
I20240506 11:39:08.984004 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391"
I20240506 11:39:08.984239 12183 raft_consensus.cc:2379] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e36d0b9df4204b1d8a524fd2a2f78042 in current term 6: Already voted for candidate f1dfb32eb83944708aa5de1a681ed391 in this term.
I20240506 11:39:08.983819 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 election: Requested vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
W20240506 11:39:08.984884 12117 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.006s (SENT) [suppressed 3 similar messages]
W20240506 11:39:08.987207 12247 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.008s (SENT) [suppressed 3 similar messages]
W20240506 11:39:08.992609 12117 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.006s (SENT)
W20240506 11:39:08.996950 12247 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.008s (SENT)
W20240506 11:39:09.000146 12117 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.008s (SENT)
I20240506 11:39:09.000284 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:09.000609 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240506 11:39:09.001482 12247 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.008s (SENT)
I20240506 11:39:09.001616 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:09.001931 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240506 11:39:09.009375 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:09.009574 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:09.010156 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:09.010401 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:09.010681 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 7 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" is_pre_election: true
I20240506 11:39:09.010959 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:09.011241 12312 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1dfb32eb83944708aa5de1a681ed391 in term 6.
I20240506 11:39:09.011734 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:39:09.011865 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:09.011960 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 7 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:09.012065 12475 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 11:39:09.012178 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 6.
I20240506 11:39:09.012388 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:09.012504 12475 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 6 FOLLOWER]: Advancing to term 7
I20240506 11:39:09.012514 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:39:09.012746 12541 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240506 11:39:09.012838 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:09.012943 12541 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 6 FOLLOWER]: Advancing to term 7
I20240506 11:39:09.014644 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:09.015189 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:09.015440 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 7 election: Requested vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:09.015889 12475 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 7 election: Requested vote from peers e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131:34507), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:09.016148 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 7 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391"
I20240506 11:39:09.016183 12312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 7 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "e36d0b9df4204b1d8a524fd2a2f78042"
I20240506 11:39:09.016356 12183 raft_consensus.cc:2379] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e36d0b9df4204b1d8a524fd2a2f78042 in current term 7: Already voted for candidate f1dfb32eb83944708aa5de1a681ed391 in this term.
I20240506 11:39:09.016396 12312 raft_consensus.cc:2379] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f1dfb32eb83944708aa5de1a681ed391 in current term 7: Already voted for candidate e36d0b9df4204b1d8a524fd2a2f78042 in this term.
W20240506 11:39:09.028278 12247 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.007s (SENT)
W20240506 11:39:09.034401 12117 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.011s (SENT)
W20240506 11:39:09.036844 12247 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.010s (SENT)
I20240506 11:39:09.036978 12247 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, f1dfb32eb83944708aa5de1a681ed391
I20240506 11:39:09.037238 12541 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20240506 11:39:09.043933 12117 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.013s (SENT)
I20240506 11:39:09.044067 12117 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1dfb32eb83944708aa5de1a681ed391; no voters: b6a0d74bcf824aebbbb86cf0fd12f16e, e36d0b9df4204b1d8a524fd2a2f78042
I20240506 11:39:09.044360 12475 raft_consensus.cc:2726] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240506 11:39:09.045984 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:09.046149 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:09.046810 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:09.047263 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 8 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391" is_pre_election: true
I20240506 11:39:09.047487 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 7.
I20240506 11:39:09.047874 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:39:09.048099 12541 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240506 11:39:09.048194 12541 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:09.048274 12541 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 7 FOLLOWER]: Advancing to term 8
I20240506 11:39:09.050043 12541 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:09.050588 12541 leader_election.cc:288] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 8 election: Requested vote from peers f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689), b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317)
I20240506 11:39:09.051115 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 8 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "f1dfb32eb83944708aa5de1a681ed391"
I20240506 11:39:09.051303 12183 raft_consensus.cc:3037] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 7 FOLLOWER]: Advancing to term 8
I20240506 11:39:09.053968 12183 raft_consensus.cc:2454] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e36d0b9df4204b1d8a524fd2a2f78042 in term 8.
I20240506 11:39:09.054353 12246 leader_election.cc:302] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e36d0b9df4204b1d8a524fd2a2f78042, f1dfb32eb83944708aa5de1a681ed391; no voters:
I20240506 11:39:09.054720 12541 raft_consensus.cc:2781] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 8 FOLLOWER]: Leader election won for term 8
I20240506 11:39:09.055238 12541 raft_consensus.cc:686] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 8 LEADER]: Becoming Leader. State: Replica: e36d0b9df4204b1d8a524fd2a2f78042, State: Running, Role: LEADER
I20240506 11:39:09.055603 12541 consensus_queue.cc:237] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 5.54, Last appended by leader: 54, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:09.058068 12183 raft_consensus.cc:1261] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 8 FOLLOWER]: Refusing update from remote peer e36d0b9df4204b1d8a524fd2a2f78042: Log matching property violated. Preceding OpId in replica: term: 5 index: 54. Preceding OpId from leader: term: 8 index: 55. (index mismatch)
I20240506 11:39:09.057873 11908 catalog_manager.cc:5484] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 reported cstate change: term changed from 5 to 8, leader changed from b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129) to e36d0b9df4204b1d8a524fd2a2f78042 (127.10.45.131). New cstate: current_term: 8 leader_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } health_report { overall_health: UNKNOWN } } }
I20240506 11:39:09.058612 12696 consensus_queue.cc:1035] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
W20240506 11:39:09.062054 12247 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.007s (SENT)
W20240506 11:39:09.081354 12247 leader_election.cc:334] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Timed out: RequestConsensusVote RPC to 127.10.45.129:39317 timed out after 0.015s (SENT)
I20240506 11:39:09.166106 12183 raft_consensus.cc:1226] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 8 FOLLOWER]: Rejecting Update request from peer b6a0d74bcf824aebbbb86cf0fd12f16e for earlier term 5. Current term is 8. Ops: []
I20240506 11:39:09.168187 12051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e"
I20240506 11:39:09.168275 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "f1dfb32eb83944708aa5de1a681ed391" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:09.168622 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d657e07d5047b98c616f9a175d5cfe" candidate_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" candidate_term: 6 candidate_status { last_received { term: 5 index: 54 } } ignore_live_leader: false dest_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" is_pre_election: true
I20240506 11:39:09.168747 12312 raft_consensus.cc:1226] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 [term 8 LEADER]: Rejecting Update request from peer b6a0d74bcf824aebbbb86cf0fd12f16e for earlier term 5. Current term is 8. Ops: []
I20240506 11:39:09.169214 12652 consensus_queue.cc:1046] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 }, Status: INVALID_TERM, Last received: 5.54, Next index: 55, Last known committed idx: 55, Time since last communication: 0.000s
I20240506 11:39:09.169389 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 11970
I20240506 11:39:09.169430 12652 consensus_queue.cc:1225] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [LEADER]: Peer e36d0b9df4204b1d8a524fd2a2f78042 log is divergent from this leader: its last log entry 8.55 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 55
I20240506 11:39:09.169550 12652 raft_consensus.cc:3032] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 5 LEADER]: Stepping down as leader of term 5
I20240506 11:39:09.169621 12652 raft_consensus.cc:729] T 63d657e07d5047b98c616f9a175d5cfe P b6a0d74bcf824aebbbb86cf0fd12f16e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b6a0d74bcf824aebbbb86cf0fd12f16e, State: Running, Role: LEADER
W20240506 11:39:09.200635 12116 connection.cc:535] server connection from 127.10.45.129:43381 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 11:39:09.200708 12247 connection.cc:535] client connection to 127.10.45.129:39317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 11:39:09.201100 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 12101
W20240506 11:39:09.202215 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:39:09.207890 12223 generic_service.cc:196] Checking for leaks (request via RPC)
W20240506 11:39:09.214188 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240506 11:39:09.225670 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240506 11:39:09.236802 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240506 11:39:09.245796 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240506 11:39:09.257184 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240506 11:39:09.268520 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240506 11:39:09.279786 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240506 11:39:09.287631 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240506 11:39:09.298154 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240506 11:39:09.309777 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240506 11:39:09.320865 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240506 11:39:09.332247 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240506 11:39:09.336898 12475 raft_consensus.cc:482] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e36d0b9df4204b1d8a524fd2a2f78042)
I20240506 11:39:09.337118 12475 raft_consensus.cc:504] T 63d657e07d5047b98c616f9a175d5cfe P f1dfb32eb83944708aa5de1a681ed391 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1dfb32eb83944708aa5de1a681ed391" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45689 } } peers { permanent_uuid: "e36d0b9df4204b1d8a524fd2a2f78042" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 34507 } } peers { permanent_uuid: "b6a0d74bcf824aebbbb86cf0fd12f16e" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39317 } }
I20240506 11:39:09.337464 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 12101
W20240506 11:39:09.345329 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240506 11:39:09.356608 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240506 11:39:09.363384 12246 connection.cc:535] client connection to 127.10.45.130:45689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 11:39:09.364194 12246 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer f1dfb32eb83944708aa5de1a681ed391 (127.10.45.130:45689): Couldn't send request to peer f1dfb32eb83944708aa5de1a681ed391. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.130:45689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:39:09.364364 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 12231
W20240506 11:39:09.368407 12247 consensus_peers.cc:485] T 63d657e07d5047b98c616f9a175d5cfe P e36d0b9df4204b1d8a524fd2a2f78042 -> Peer b6a0d74bcf824aebbbb86cf0fd12f16e (127.10.45.129:39317): Couldn't send request to peer b6a0d74bcf824aebbbb86cf0fd12f16e. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:39317: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240506 11:39:09.371744 12353 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:39:09.492468 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 12231
I20240506 11:39:09.516232 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 11880
I20240506 11:39:09.517264 11941 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:39:09.622390 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 11880
2024-05-06T11:39:09Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElections (34333 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency
I20240506 11:39:09.666126 10422 test_util.cc:268] Using random seed: -2071495880
I20240506 11:39:09.667949 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:39:09.668012 10422 ts_itest-base.cc:113] --------------
I20240506 11:39:09.668069 10422 ts_itest-base.cc:114] 3 tablet servers
I20240506 11:39:09.668120 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:39:09.668181 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:39:09Z Disabled control of system clock
I20240506 11:39:09.692905 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:43667
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:43667 with env {}
W20240506 11:39:10.008778 12714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:39:10.009050 12714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:39:10.009183 12714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:39:10.016275 12714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:39:10.016374 12714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:39:10.016446 12714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:39:10.016522 12714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:39:10.025219 12714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:43667
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:43667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:39:10.026456 12714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:39:10.028133 12714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:39:10.039075 12719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:10.039963 12720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:10.040812 12722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:39:10.041666 12714 server_base.cc:1034] running on GCE node
I20240506 11:39:10.042346 12714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:39:10.043706 12714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:39:10.044911 12714 hybrid_clock.cc:648] HybridClock initialized: now 1714995550044857 us; error 30 us; skew 500 ppm
I20240506 11:39:10.045245 12714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:39:10.047412 12714 webserver.cc:460] Webserver started at http://127.10.45.190:35147/ using document root <none> and password file <none>
I20240506 11:39:10.048090 12714 fs_manager.cc:359] Metadata directory not provided
I20240506 11:39:10.048208 12714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:39:10.048501 12714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:39:10.050879 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7851e30a07e642cc844a5c32667d8ce7"
format_stamp: "Formatted at 2024-05-06 11:39:10 on dist-test-slave-7l0g"
I20240506 11:39:10.051623 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7851e30a07e642cc844a5c32667d8ce7"
format_stamp: "Formatted at 2024-05-06 11:39:10 on dist-test-slave-7l0g"
I20240506 11:39:10.056190 12714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:39:10.059286 12728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:10.060420 12714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240506 11:39:10.060572 12714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7851e30a07e642cc844a5c32667d8ce7"
format_stamp: "Formatted at 2024-05-06 11:39:10 on dist-test-slave-7l0g"
I20240506 11:39:10.060740 12714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:39:10.085438 12714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:39:10.086149 12714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:39:10.086424 12714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:39:10.102916 12714 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:43667
I20240506 11:39:10.102970 12779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:43667 every 8 connection(s)
I20240506 11:39:10.104178 12714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:39:10.107550 12780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:39:10.108333 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 12714
I20240506 11:39:10.108527 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:39:10.119372 12780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7: Bootstrap starting.
I20240506 11:39:10.122327 12780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7: Neither blocks nor log segments found. Creating new log.
I20240506 11:39:10.123257 12780 log.cc:824] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7: Log is configured to *not* fsync() on all Append() calls
I20240506 11:39:10.125602 12780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7: No bootstrap required, opened a new log
I20240506 11:39:10.131287 12780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7851e30a07e642cc844a5c32667d8ce7" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 43667 } }
I20240506 11:39:10.131603 12780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:39:10.131703 12780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7851e30a07e642cc844a5c32667d8ce7, State: Initialized, Role: FOLLOWER
I20240506 11:39:10.132264 12780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7851e30a07e642cc844a5c32667d8ce7" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 43667 } }
I20240506 11:39:10.132462 12780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:39:10.132596 12780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:39:10.132743 12780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:39:10.134696 12780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7851e30a07e642cc844a5c32667d8ce7" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 43667 } }
I20240506 11:39:10.135183 12780 leader_election.cc:302] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7851e30a07e642cc844a5c32667d8ce7; no voters:
I20240506 11:39:10.135737 12780 leader_election.cc:288] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:39:10.135852 12785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:39:10.136451 12785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [term 1 LEADER]: Becoming Leader. State: Replica: 7851e30a07e642cc844a5c32667d8ce7, State: Running, Role: LEADER
I20240506 11:39:10.136978 12785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7851e30a07e642cc844a5c32667d8ce7" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 43667 } }
I20240506 11:39:10.137542 12780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:39:10.139505 12787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7851e30a07e642cc844a5c32667d8ce7. Latest consensus state: current_term: 1 leader_uuid: "7851e30a07e642cc844a5c32667d8ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7851e30a07e642cc844a5c32667d8ce7" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 43667 } } }
I20240506 11:39:10.139914 12787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [sys.catalog]: This master's current role is: LEADER
I20240506 11:39:10.142231 12793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:39:10.142069 12786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7851e30a07e642cc844a5c32667d8ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7851e30a07e642cc844a5c32667d8ce7" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 43667 } } }
I20240506 11:39:10.142362 12786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7 [sys.catalog]: This master's current role is: LEADER
I20240506 11:39:10.147126 12793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:39:10.155668 12793 catalog_manager.cc:1347] Generated new cluster ID: 652aaf137ba049f2a013b105f2ed93d9
I20240506 11:39:10.155772 12793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:39:10.168141 12793 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:39:10.169205 12793 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:39:10.179020 12793 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7851e30a07e642cc844a5c32667d8ce7: Generated new TSK 0
I20240506 11:39:10.179780 12793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:39:10.186364 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:43667
--builtin_ntp_servers=127.10.45.148:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
W20240506 11:39:10.534677 12804 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20240506 11:39:10.534998 12804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:39:10.535109 12804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:39:10.535238 12804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:39:10.542562 12804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:39:10.542763 12804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:39:10.551332 12804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.45.190:43667
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:39:10.552567 12804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:39:10.554181 12804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:39:10.565970 12812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:10.566812 12810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:10.567644 12809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:39:10.568431 12804 server_base.cc:1034] running on GCE node
I20240506 11:39:10.568964 12804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:39:10.569777 12804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:39:10.571015 12804 hybrid_clock.cc:648] HybridClock initialized: now 1714995550570967 us; error 33 us; skew 500 ppm
I20240506 11:39:10.571271 12804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:39:10.573236 12804 webserver.cc:460] Webserver started at http://127.10.45.129:39055/ using document root <none> and password file <none>
I20240506 11:39:10.573895 12804 fs_manager.cc:359] Metadata directory not provided
I20240506 11:39:10.573983 12804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:39:10.574240 12804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:39:10.576535 12804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
format_stamp: "Formatted at 2024-05-06 11:39:10 on dist-test-slave-7l0g"
I20240506 11:39:10.577195 12804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
format_stamp: "Formatted at 2024-05-06 11:39:10 on dist-test-slave-7l0g"
I20240506 11:39:10.581504 12804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240506 11:39:10.584553 12818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:10.585717 12804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240506 11:39:10.585837 12804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
format_stamp: "Formatted at 2024-05-06 11:39:10 on dist-test-slave-7l0g"
I20240506 11:39:10.585987 12804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:39:10.607805 12804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:39:10.608534 12804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:39:10.608753 12804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:39:10.609575 12804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:39:10.610837 12804 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:39:10.610914 12804 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:10.611027 12804 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:39:10.611100 12804 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:10.640504 12804 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:33081
I20240506 11:39:10.640548 12930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:33081 every 8 connection(s)
I20240506 11:39:10.641675 12804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:39:10.646458 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 12804
I20240506 11:39:10.646646 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:39:10.649449 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:43667
--builtin_ntp_servers=127.10.45.148:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20240506 11:39:10.653867 12931 heartbeater.cc:347] Connected to a master server at 127.10.45.190:43667
I20240506 11:39:10.654152 12931 heartbeater.cc:464] Registering TS with master...
I20240506 11:39:10.654897 12931 heartbeater.cc:511] Master 127.10.45.190:43667 requested a full tablet report, sending...
I20240506 11:39:10.656766 12745 ts_manager.cc:194] Registered new tserver with Master: 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:10.658106 12745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:56989
W20240506 11:39:10.958103 12935 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20240506 11:39:10.958401 12935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:39:10.958492 12935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:39:10.958606 12935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:39:10.965948 12935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:39:10.966137 12935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:39:10.974749 12935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.45.190:43667
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:39:10.976087 12935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:39:10.977767 12935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:39:10.989560 12940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:10.990466 12941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:10.991284 12943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:39:10.992242 12935 server_base.cc:1034] running on GCE node
I20240506 11:39:10.992849 12935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:39:10.993826 12935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:39:10.995033 12935 hybrid_clock.cc:648] HybridClock initialized: now 1714995550994989 us; error 35 us; skew 500 ppm
I20240506 11:39:10.995337 12935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:39:10.997485 12935 webserver.cc:460] Webserver started at http://127.10.45.130:45935/ using document root <none> and password file <none>
I20240506 11:39:10.998191 12935 fs_manager.cc:359] Metadata directory not provided
I20240506 11:39:10.998306 12935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:39:10.998616 12935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:39:11.001222 12935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "30ffd8145bbd483bbb01f029813c4541"
format_stamp: "Formatted at 2024-05-06 11:39:10 on dist-test-slave-7l0g"
I20240506 11:39:11.001963 12935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "30ffd8145bbd483bbb01f029813c4541"
format_stamp: "Formatted at 2024-05-06 11:39:10 on dist-test-slave-7l0g"
I20240506 11:39:11.008049 12935 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240506 11:39:11.011304 12949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:11.012606 12935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240506 11:39:11.012760 12935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "30ffd8145bbd483bbb01f029813c4541"
format_stamp: "Formatted at 2024-05-06 11:39:10 on dist-test-slave-7l0g"
I20240506 11:39:11.012946 12935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:39:11.035714 12935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:39:11.036562 12935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:39:11.036845 12935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:39:11.037818 12935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:39:11.039413 12935 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:39:11.039501 12935 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:11.039587 12935 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:39:11.039646 12935 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:11.069559 12935 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:41551
I20240506 11:39:11.069602 13061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:41551 every 8 connection(s)
I20240506 11:39:11.070817 12935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:39:11.077616 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 12935
I20240506 11:39:11.077831 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:39:11.080787 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:43667
--builtin_ntp_servers=127.10.45.148:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20240506 11:39:11.083406 13062 heartbeater.cc:347] Connected to a master server at 127.10.45.190:43667
I20240506 11:39:11.083775 13062 heartbeater.cc:464] Registering TS with master...
I20240506 11:39:11.084758 13062 heartbeater.cc:511] Master 127.10.45.190:43667 requested a full tablet report, sending...
I20240506 11:39:11.086300 12745 ts_manager.cc:194] Registered new tserver with Master: 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551)
I20240506 11:39:11.087394 12745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:59159
W20240506 11:39:11.390411 13065 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20240506 11:39:11.390744 13065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:39:11.390849 13065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:39:11.390956 13065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:39:11.398053 13065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:39:11.398224 13065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:39:11.406683 13065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.45.190:43667
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:39:11.407943 13065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:39:11.409536 13065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:39:11.421500 13071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:11.422360 13073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:11.423259 13070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:39:11.424221 13065 server_base.cc:1034] running on GCE node
I20240506 11:39:11.424834 13065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:39:11.425779 13065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:39:11.427013 13065 hybrid_clock.cc:648] HybridClock initialized: now 1714995551426976 us; error 26 us; skew 500 ppm
I20240506 11:39:11.427315 13065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:39:11.429672 13065 webserver.cc:460] Webserver started at http://127.10.45.131:34127/ using document root <none> and password file <none>
I20240506 11:39:11.430374 13065 fs_manager.cc:359] Metadata directory not provided
I20240506 11:39:11.430488 13065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:39:11.430799 13065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:39:11.433384 13065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "89b865272ad043919295f14ca4025cf6"
format_stamp: "Formatted at 2024-05-06 11:39:11 on dist-test-slave-7l0g"
I20240506 11:39:11.434142 13065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "89b865272ad043919295f14ca4025cf6"
format_stamp: "Formatted at 2024-05-06 11:39:11 on dist-test-slave-7l0g"
I20240506 11:39:11.439214 13065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240506 11:39:11.442356 13079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:11.443692 13065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240506 11:39:11.443850 13065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "89b865272ad043919295f14ca4025cf6"
format_stamp: "Formatted at 2024-05-06 11:39:11 on dist-test-slave-7l0g"
I20240506 11:39:11.444061 13065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:39:11.456763 13065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:39:11.457551 13065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:39:11.457821 13065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:39:11.458721 13065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:39:11.460150 13065 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:39:11.460249 13065 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:11.460358 13065 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:39:11.460427 13065 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:11.491180 13065 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:32893
I20240506 11:39:11.491230 13191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:32893 every 8 connection(s)
I20240506 11:39:11.492411 13065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:39:11.499912 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 13065
I20240506 11:39:11.500093 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:39:11.504334 13192 heartbeater.cc:347] Connected to a master server at 127.10.45.190:43667
I20240506 11:39:11.504591 13192 heartbeater.cc:464] Registering TS with master...
I20240506 11:39:11.505322 13192 heartbeater.cc:511] Master 127.10.45.190:43667 requested a full tablet report, sending...
I20240506 11:39:11.506512 12745 ts_manager.cc:194] Registered new tserver with Master: 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893)
I20240506 11:39:11.507306 12745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:57521
I20240506 11:39:11.515460 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:39:11.518381 10422 test_util.cc:268] Using random seed: -2069643638
I20240506 11:39:11.534457 12745 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44082:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 11:39:11.535873 12745 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 11:39:11.559394 13127 tablet_service.cc:1467] Processing CreateTablet for tablet 831a913fdab244808c7b4bae4a4a56c3 (DEFAULT_TABLE table=test-workload [id=afea6e0404bc49e9a1bad811c6f3c29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:39:11.559844 12997 tablet_service.cc:1467] Processing CreateTablet for tablet 831a913fdab244808c7b4bae4a4a56c3 (DEFAULT_TABLE table=test-workload [id=afea6e0404bc49e9a1bad811c6f3c29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:39:11.560621 13127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831a913fdab244808c7b4bae4a4a56c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:39:11.561193 12997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831a913fdab244808c7b4bae4a4a56c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:39:11.561261 12866 tablet_service.cc:1467] Processing CreateTablet for tablet 831a913fdab244808c7b4bae4a4a56c3 (DEFAULT_TABLE table=test-workload [id=afea6e0404bc49e9a1bad811c6f3c29a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:39:11.562878 12866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831a913fdab244808c7b4bae4a4a56c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:39:11.570665 13210 tablet_bootstrap.cc:492] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6: Bootstrap starting.
I20240506 11:39:11.573348 13211 tablet_bootstrap.cc:492] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541: Bootstrap starting.
I20240506 11:39:11.574260 13210 tablet_bootstrap.cc:654] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6: Neither blocks nor log segments found. Creating new log.
I20240506 11:39:11.575464 13212 tablet_bootstrap.cc:492] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab: Bootstrap starting.
I20240506 11:39:11.575810 13210 log.cc:824] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6: Log is configured to *not* fsync() on all Append() calls
I20240506 11:39:11.576730 13211 tablet_bootstrap.cc:654] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541: Neither blocks nor log segments found. Creating new log.
I20240506 11:39:11.577796 13211 log.cc:824] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541: Log is configured to *not* fsync() on all Append() calls
I20240506 11:39:11.578411 13212 tablet_bootstrap.cc:654] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab: Neither blocks nor log segments found. Creating new log.
I20240506 11:39:11.578934 13210 tablet_bootstrap.cc:492] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6: No bootstrap required, opened a new log
I20240506 11:39:11.579219 13210 ts_tablet_manager.cc:1367] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240506 11:39:11.579578 13212 log.cc:824] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab: Log is configured to *not* fsync() on all Append() calls
I20240506 11:39:11.580276 13211 tablet_bootstrap.cc:492] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541: No bootstrap required, opened a new log
I20240506 11:39:11.580513 13211 ts_tablet_manager.cc:1367] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240506 11:39:11.581424 13212 tablet_bootstrap.cc:492] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab: No bootstrap required, opened a new log
I20240506 11:39:11.581620 13212 ts_tablet_manager.cc:1367] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240506 11:39:11.587568 13212 raft_consensus.cc:348] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.587504 13210 raft_consensus.cc:348] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.587869 13212 raft_consensus.cc:374] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:39:11.587877 13210 raft_consensus.cc:374] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:39:11.587961 13210 raft_consensus.cc:729] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b865272ad043919295f14ca4025cf6, State: Initialized, Role: FOLLOWER
I20240506 11:39:11.587976 13212 raft_consensus.cc:729] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d74671f77b041ae90c1cb2bc6d6c5ab, State: Initialized, Role: FOLLOWER
I20240506 11:39:11.588524 13212 consensus_queue.cc:260] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.588670 13210 consensus_queue.cc:260] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.588651 13211 raft_consensus.cc:348] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.589006 13211 raft_consensus.cc:374] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:39:11.589159 13211 raft_consensus.cc:729] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ffd8145bbd483bbb01f029813c4541, State: Initialized, Role: FOLLOWER
I20240506 11:39:11.589510 12931 heartbeater.cc:503] Master 127.10.45.190:43667 was elected leader, sending a full tablet report...
I20240506 11:39:11.590183 13212 ts_tablet_manager.cc:1398] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240506 11:39:11.591702 13211 consensus_queue.cc:260] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.592708 13219 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:11.592787 13062 heartbeater.cc:503] Master 127.10.45.190:43667 was elected leader, sending a full tablet report...
I20240506 11:39:11.592967 13219 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.598639 13217 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:11.599288 13211 ts_tablet_manager.cc:1398] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541: Time spent starting tablet: real 0.019s user 0.011s sys 0.001s
I20240506 11:39:11.600045 13216 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:11.598927 13192 heartbeater.cc:503] Master 127.10.45.190:43667 was elected leader, sending a full tablet report...
I20240506 11:39:11.600397 13217 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.600546 13216 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.602123 13217 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:11.599861 13210 ts_tablet_manager.cc:1398] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6: Time spent starting tablet: real 0.021s user 0.012s sys 0.004s
I20240506 11:39:11.604328 13219 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:11.608834 13216 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551)
W20240506 11:39:11.611428 12953 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.45.129:33081 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
W20240506 11:39:11.611850 12819 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.45.131:32893 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20240506 11:39:11.625012 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:11.625499 13147 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d74671f77b041ae90c1cb2bc6d6c5ab in term 0.
I20240506 11:39:11.625512 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:11.625849 13146 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ffd8145bbd483bbb01f029813c4541 in term 0.
I20240506 11:39:11.626147 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:11.626292 12819 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:11.626438 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:11.626623 12886 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ffd8145bbd483bbb01f029813c4541 in term 0.
I20240506 11:39:11.626785 13219 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 11:39:11.626921 13219 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:11.627069 13219 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:39:11.627318 13216 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 11:39:11.627502 13216 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:11.627640 13216 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:39:11.631134 13219 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.631431 13216 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.631991 13219 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 1 election: Requested vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
W20240506 11:39:11.632267 12822 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551): Timed out: connection negotiation to 127.10.45.130:41551 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20240506 11:39:11.632472 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6"
I20240506 11:39:11.632647 13146 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:39:11.632705 13216 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [CANDIDATE]: Term 1 election: Requested vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551)
I20240506 11:39:11.633363 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
I20240506 11:39:11.633628 12886 raft_consensus.cc:2379] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ffd8145bbd483bbb01f029813c4541 in current term 1: Already voted for candidate 4d74671f77b041ae90c1cb2bc6d6c5ab in this term.
W20240506 11:39:11.635082 13083 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.45.130:41551 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20240506 11:39:11.635551 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541"
I20240506 11:39:11.635967 13146 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d74671f77b041ae90c1cb2bc6d6c5ab in term 1.
I20240506 11:39:11.636178 13017 raft_consensus.cc:2379] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d74671f77b041ae90c1cb2bc6d6c5ab in current term 1: Already voted for candidate 30ffd8145bbd483bbb01f029813c4541 in this term.
I20240506 11:39:11.636381 12819 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:11.636727 13216 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:39:11.639178 13216 raft_consensus.cc:686] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 1 LEADER]: Becoming Leader. State: Replica: 4d74671f77b041ae90c1cb2bc6d6c5ab, State: Running, Role: LEADER
I20240506 11:39:11.639710 13216 consensus_queue.cc:237] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.641978 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:11.642724 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6"
I20240506 11:39:11.643021 13146 raft_consensus.cc:2379] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ffd8145bbd483bbb01f029813c4541 in current term 1: Already voted for candidate 4d74671f77b041ae90c1cb2bc6d6c5ab in this term.
I20240506 11:39:11.643499 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:11.643743 13017 raft_consensus.cc:2379] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89b865272ad043919295f14ca4025cf6 in current term 1: Already voted for candidate 30ffd8145bbd483bbb01f029813c4541 in this term.
W20240506 11:39:11.643716 12932 tablet.cc:2369] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:39:11.644194 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:11.644603 13218 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 11:39:11.645076 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:11.645602 13219 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240506 11:39:11.650233 13216 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240506 11:39:11.650375 13216 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240506 11:39:11.651046 12745 catalog_manager.cc:5484] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129). New cstate: current_term: 1 leader_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } health_report { overall_health: HEALTHY } } }
I20240506 11:39:11.662302 13219 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:11.662490 13219 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.663477 13219 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:11.663774 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:11.664183 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:11.664783 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:11.682634 13219 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:11.685688 13218 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:11.685889 13218 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:11.687037 13218 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:11.688100 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:11.688338 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:11.688858 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:11.689150 13218 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 11:39:12.214802 13146 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240506 11:39:12.214802 13016 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240506 11:39:12.214911 13146 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20240506 11:39:12.214922 13016 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240506 11:39:12.546104 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:12.546378 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:12.547696 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:12.548038 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:12.548422 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:12.548936 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
W20240506 11:39:12.582655 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.742s user 0.001s sys 0.000s
W20240506 11:39:12.582672 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.743s user 0.000s sys 0.000s
I20240506 11:39:12.805943 13244 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:12.806190 13244 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:12.808037 13244 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:12.807952 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:12.808557 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:12.809304 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:12.815399 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:12.849218 13248 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=0, attempt_no=0}) took 1149 ms. Trace:
I20240506 11:39:12.849411 13248 rpcz_store.cc:276] 0506 11:39:11.699782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:11.699843 (+ 61us) service_pool.cc:225] Handling call
0506 11:39:12.849152 (+1149309us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:12.212976 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:12.213109 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:12.213116 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:39:12.213118 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:12.213212 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:12.213247 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:12.213251 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:12.213307 (+ 56us) tablet.cc:699] Row locks acquired
0506 11:39:12.213309 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:12.213351 (+ 42us) write_op.cc:270] Start()
0506 11:39:12.361520 (+148169us) write_op.cc:276] Timestamp: P: 1714995552213319 usec, L: 0
0506 11:39:12.361525 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:12.361754 (+ 229us) log.cc:842] Serialized 345 byte log entry
0506 11:39:12.848413 (+486659us) op_driver.cc:444] REPLICATION: finished
0506 11:39:12.848637 (+ 224us) write_op.cc:301] APPLY: starting
0506 11:39:12.848662 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:12.848721 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:12.848764 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:12.848836 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:12.848865 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:12.848871 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:39:12.848954 (+ 83us) log.cc:842] Serialized 22 byte log entry
0506 11:39:12.849024 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:12.849040 (+ 16us) write_op.cc:454] Released schema lock
0506 11:39:12.849047 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1778176,"child_traces":[["op",{"apply.queue_time_us":77,"mutex_wait_us":33,"num_ops":1,"prepare.queue_time_us":509427,"prepare.run_cpu_time_us":780,"prepare.run_wall_time_us":148847,"replication_time_us":486848}]]}
I20240506 11:39:13.102540 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:13.102759 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:13.103564 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:13.104126 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:13.105145 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:13.105616 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:13.105818 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:13.730060 13244 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:14.071563 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:14.071733 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:14.072481 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:14.072945 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:14.072943 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:14.073521 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:14.073799 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:14.081200 13259 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=3, attempt_no=0}) took 1229 ms. Trace:
I20240506 11:39:14.081328 13259 rpcz_store.cc:276] 0506 11:39:12.851927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:12.852017 (+ 90us) service_pool.cc:225] Handling call
0506 11:39:14.081141 (+1229124us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:12.852676 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:12.852784 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:12.852788 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:12.852790 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:12.852875 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:12.852882 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:12.852885 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:12.852921 (+ 36us) tablet.cc:699] Row locks acquired
0506 11:39:12.852922 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:12.852952 (+ 30us) write_op.cc:270] Start()
0506 11:39:13.354076 (+501124us) write_op.cc:276] Timestamp: P: 1714995552852910 usec, L: 0
0506 11:39:13.354080 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:13.354331 (+ 251us) log.cc:842] Serialized 354 byte log entry
0506 11:39:14.080343 (+726012us) op_driver.cc:444] REPLICATION: finished
0506 11:39:14.080729 (+ 386us) write_op.cc:301] APPLY: starting
0506 11:39:14.080744 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:14.080774 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:14.080800 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:14.080855 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:14.080877 (+ 22us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:14.080881 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:14.080948 (+ 67us) log.cc:842] Serialized 22 byte log entry
0506 11:39:14.081030 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:14.081049 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:14.081055 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":315,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":667,"prepare.run_wall_time_us":501730,"replication_time_us":726235,"thread_start_us":215,"threads_started":1}]]}
W20240506 11:39:14.083082 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.822s user 0.001s sys 0.000s
W20240506 11:39:14.083250 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.822s user 0.001s sys 0.000s
I20240506 11:39:14.565387 13244 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:14.565604 13244 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:14.566272 13244 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:14.566737 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:14.566803 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:14.567392 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:14.994974 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:14.995177 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:14.995829 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:14.996305 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:14.996310 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:14.996874 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:14.997267 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:15.012037 13265 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=4, attempt_no=0}) took 1901 ms. Trace:
I20240506 11:39:15.012144 13265 rpcz_store.cc:276] 0506 11:39:13.110528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:13.110732 (+ 204us) service_pool.cc:225] Handling call
0506 11:39:15.011987 (+1901255us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:13.354454 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:13.354558 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:13.354563 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:13.354565 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:13.354650 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:13.354661 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:13.354665 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:13.354715 (+ 50us) tablet.cc:699] Row locks acquired
0506 11:39:13.354717 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:13.354775 (+ 58us) write_op.cc:270] Start()
0506 11:39:14.077927 (+723152us) write_op.cc:276] Timestamp: P: 1714995553354726 usec, L: 0
0506 11:39:14.077932 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:14.078135 (+ 203us) log.cc:842] Serialized 354 byte log entry
0506 11:39:15.011149 (+933014us) op_driver.cc:444] REPLICATION: finished
0506 11:39:15.011568 (+ 419us) write_op.cc:301] APPLY: starting
0506 11:39:15.011582 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:15.011615 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:15.011646 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:15.011730 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:15.011754 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:15.011758 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:15.011832 (+ 74us) log.cc:842] Serialized 22 byte log entry
0506 11:39:15.011894 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:15.011908 (+ 14us) write_op.cc:454] Released schema lock
0506 11:39:15.011914 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":365,"num_ops":1,"prepare.queue_time_us":243318,"prepare.run_cpu_time_us":686,"prepare.run_wall_time_us":723758,"replication_time_us":933189,"thread_start_us":277,"threads_started":1}]]}
I20240506 11:39:15.255345 13244 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:15.285010 13265 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=5, attempt_no=0}) took 1201 ms. Trace:
I20240506 11:39:15.285181 13265 rpcz_store.cc:276] 0506 11:39:14.083367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:14.083425 (+ 58us) service_pool.cc:225] Handling call
0506 11:39:15.284947 (+1201522us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:14.083834 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:14.083904 (+ 70us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:14.083907 (+ 3us) write_op.cc:435] Acquired schema lock
0506 11:39:14.083908 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:14.083967 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:14.083975 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:14.083978 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:14.084011 (+ 33us) tablet.cc:699] Row locks acquired
0506 11:39:14.084012 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:14.084038 (+ 26us) write_op.cc:270] Start()
0506 11:39:14.541189 (+457151us) write_op.cc:276] Timestamp: P: 1714995554083975 usec, L: 0
0506 11:39:14.541193 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:14.541494 (+ 301us) log.cc:842] Serialized 354 byte log entry
0506 11:39:15.284339 (+742845us) op_driver.cc:444] REPLICATION: finished
0506 11:39:15.284449 (+ 110us) write_op.cc:301] APPLY: starting
0506 11:39:15.284468 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:15.284523 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:15.284548 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:15.284613 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:15.284639 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:15.284644 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:15.284714 (+ 70us) log.cc:842] Serialized 22 byte log entry
0506 11:39:15.284812 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:15.284841 (+ 29us) write_op.cc:454] Released schema lock
0506 11:39:15.284848 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":457717,"replication_time_us":743121}]]}
I20240506 11:39:15.286664 12845 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 12) took 1052 ms. Trace:
I20240506 11:39:15.286769 12845 rpcz_store.cc:276] 0506 11:39:14.234406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:14.234470 (+ 64us) service_pool.cc:225] Handling call
0506 11:39:14.234714 (+ 244us) tablet_service.cc:2780] Created scanner 5c0ffcc03d81417fb41a72769560521a for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 1ebf19da59a342cfb1a73d197ae4e085
0506 11:39:14.235673 (+ 959us) tablet_service.cc:2920] Creating iterator
0506 11:39:14.235723 (+ 50us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:14.541373 (+305650us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:15.284860 (+743487us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1049122 microseconds
0506 11:39:15.284954 (+ 94us) tablet_service.cc:2945] Iterator created
0506 11:39:15.285548 (+ 594us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:15.285583 (+ 35us) tablet_service.cc:3010] has_more: true
0506 11:39:15.285651 (+ 68us) tablet_service.cc:3027] Continuing scan request
0506 11:39:15.285726 (+ 75us) tablet_service.cc:3091] Found scanner 5c0ffcc03d81417fb41a72769560521a for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 1ebf19da59a342cfb1a73d197ae4e085
0506 11:39:15.286591 (+ 865us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":11136}
I20240506 11:39:15.287411 12846 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 13) took 1052 ms. Trace:
I20240506 11:39:15.287506 12846 rpcz_store.cc:276] 0506 11:39:14.234533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:14.234597 (+ 64us) service_pool.cc:225] Handling call
0506 11:39:14.234739 (+ 142us) tablet_service.cc:2780] Created scanner fb430e4259ef46ca82dc4949d13937c4 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 9d3a5aafb4214e9a9319120c4a695a56
0506 11:39:14.235673 (+ 934us) tablet_service.cc:2920] Creating iterator
0506 11:39:14.235726 (+ 53us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:14.541383 (+305657us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:15.285867 (+744484us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1050126 microseconds
0506 11:39:15.285955 (+ 88us) tablet_service.cc:2945] Iterator created
0506 11:39:15.286403 (+ 448us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:15.286431 (+ 28us) tablet_service.cc:3010] has_more: true
0506 11:39:15.286455 (+ 24us) tablet_service.cc:3027] Continuing scan request
0506 11:39:15.286506 (+ 51us) tablet_service.cc:3091] Found scanner fb430e4259ef46ca82dc4949d13937c4 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 9d3a5aafb4214e9a9319120c4a695a56
0506 11:39:15.287355 (+ 849us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":44800}
W20240506 11:39:15.287855 13239 scanner-internal.cc:458] Time spent opening tablet: real 1.054s user 0.001s sys 0.000s
W20240506 11:39:15.288081 13240 scanner-internal.cc:458] Time spent opening tablet: real 1.054s user 0.001s sys 0.000s
I20240506 11:39:15.909252 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:15.909454 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:15.910303 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:15.910765 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:15.910782 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:15.911525 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:16.082639 13244 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:16.082826 13244 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:16.083634 13244 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:16.084122 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:16.084198 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:16.084746 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:16.106473 13275 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=6, attempt_no=0}) took 1092 ms. Trace:
I20240506 11:39:16.106617 13275 rpcz_store.cc:276] 0506 11:39:15.013580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:15.013650 (+ 70us) service_pool.cc:225] Handling call
0506 11:39:16.106422 (+1092772us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:15.014140 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:15.014242 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:15.014246 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:15.014247 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:15.014318 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:15.014325 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:15.014328 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:15.014363 (+ 35us) tablet.cc:699] Row locks acquired
0506 11:39:15.014364 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:15.014396 (+ 32us) write_op.cc:270] Start()
0506 11:39:15.396539 (+382143us) write_op.cc:276] Timestamp: P: 1714995555014344 usec, L: 0
0506 11:39:15.396545 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:39:15.396754 (+ 209us) log.cc:842] Serialized 354 byte log entry
0506 11:39:16.105183 (+708429us) op_driver.cc:444] REPLICATION: finished
0506 11:39:16.105772 (+ 589us) write_op.cc:301] APPLY: starting
0506 11:39:16.105791 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:16.105833 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:16.105905 (+ 72us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:16.106032 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:16.106065 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:16.106073 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:39:16.106164 (+ 91us) log.cc:842] Serialized 22 byte log entry
0506 11:39:16.106281 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:16.106300 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:16.106308 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":511,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":662,"prepare.run_wall_time_us":382733,"raft.queue_time_us":34,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":708606,"spinlock_wait_cycles":66816,"thread_start_us":376,"threads_started":1}]]}
I20240506 11:39:16.332899 13244 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:16.370674 13275 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=7, attempt_no=0}) took 1083 ms. Trace:
I20240506 11:39:16.370828 13275 rpcz_store.cc:276] 0506 11:39:15.286897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:15.286966 (+ 69us) service_pool.cc:225] Handling call
0506 11:39:16.370602 (+1083636us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:15.396924 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:15.397022 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:15.397029 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:39:15.397031 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:15.397177 (+ 146us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:15.397191 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:15.397197 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:15.397251 (+ 54us) tablet.cc:699] Row locks acquired
0506 11:39:15.397253 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:15.397298 (+ 45us) write_op.cc:270] Start()
0506 11:39:15.561457 (+164159us) write_op.cc:276] Timestamp: P: 1714995555397256 usec, L: 0
0506 11:39:15.561461 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:15.561784 (+ 323us) log.cc:842] Serialized 354 byte log entry
0506 11:39:16.369961 (+808177us) op_driver.cc:444] REPLICATION: finished
0506 11:39:16.370078 (+ 117us) write_op.cc:301] APPLY: starting
0506 11:39:16.370095 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:16.370131 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:16.370172 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:16.370233 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:16.370265 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:16.370274 (+ 9us) write_op.cc:312] APPLY: finished
0506 11:39:16.370362 (+ 88us) log.cc:842] Serialized 22 byte log entry
0506 11:39:16.370470 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:16.370490 (+ 20us) write_op.cc:454] Released schema lock
0506 11:39:16.370498 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"num_ops":1,"prepare.queue_time_us":109546,"prepare.run_cpu_time_us":842,"prepare.run_wall_time_us":164924,"replication_time_us":808465}]]}
W20240506 11:39:16.372642 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.934s user 0.001s sys 0.000s
W20240506 11:39:16.373859 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.935s user 0.001s sys 0.000s
I20240506 11:39:16.912205 13244 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:16.912412 13244 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:16.913175 13244 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:16.913661 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:16.914446 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:16.915025 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:17.028568 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:17.029075 13016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:39005 (request call id 616) took 1118 ms. Trace:
I20240506 11:39:17.029242 13016 rpcz_store.cc:276] 0506 11:39:15.911054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:15.911131 (+ 77us) service_pool.cc:225] Handling call
0506 11:39:15.911329 (+ 198us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 11:39:15.911452 (+ 123us) raft_consensus.cc:1520] Early marking committed up to index 7
0506 11:39:15.911805 (+ 353us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 11:39:15.911933 (+ 128us) write_op.cc:270] Start()
0506 11:39:16.263045 (+351112us) write_op.cc:276] Timestamp: P: 1714995555014344 usec, L: 0
0506 11:39:16.263411 (+ 366us) write_op.cc:270] Start()
0506 11:39:17.027562 (+764151us) write_op.cc:276] Timestamp: P: 1714995555397256 usec, L: 0
0506 11:39:17.028151 (+ 589us) log.cc:842] Serialized 708 byte log entry
0506 11:39:17.028441 (+ 290us) raft_consensus.cc:1631] Marking committed up to 7
0506 11:39:17.028451 (+ 10us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:17.028491 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:17.028972 (+ 481us) raft_consensus.cc:1671] finished
0506 11:39:17.028976 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:17.029031 (+ 55us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:16.263260 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:16.263360 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:16.263382 (+ 22us) write_op.cc:435] Acquired schema lock
0506 11:39:16.263383 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:16.263456 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:16.263469 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:16.263472 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:16.263514 (+ 42us) tablet.cc:699] Row locks acquired
0506 11:39:16.263515 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0506 11:39:17.028006 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:17.028117 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:17.028121 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:17.028123 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:17.028199 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:17.028237 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:17.028240 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:17.028310 (+ 70us) tablet.cc:699] Row locks acquired
0506 11:39:17.028312 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":215,"threads_started":1,"wal-append.queue_time_us":293,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":337,"prepare.run_wall_time_us":338}],["op",{"num_ops":1,"partition_lock_wait_us":31,"prepare.queue_time_us":317,"prepare.run_cpu_time_us":388,"prepare.run_wall_time_us":388,"thread_start_us":210,"threads_started":1}]]}
I20240506 11:39:17.545034 13244 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:17.595567 13289 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=9, attempt_no=0}) took 1221 ms. Trace:
I20240506 11:39:17.595697 13289 rpcz_store.cc:276] 0506 11:39:16.374447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:16.374509 (+ 62us) service_pool.cc:225] Handling call
0506 11:39:17.595513 (+1221004us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:16.534138 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:16.534229 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:16.534233 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:16.534235 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:16.534350 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:16.534360 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:16.534364 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:16.534413 (+ 49us) tablet.cc:699] Row locks acquired
0506 11:39:16.534415 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:16.534456 (+ 41us) write_op.cc:270] Start()
0506 11:39:16.898598 (+364142us) write_op.cc:276] Timestamp: P: 1714995556534402 usec, L: 0
0506 11:39:16.898603 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:16.898830 (+ 227us) log.cc:842] Serialized 354 byte log entry
0506 11:39:17.594474 (+695644us) op_driver.cc:444] REPLICATION: finished
0506 11:39:17.594997 (+ 523us) write_op.cc:301] APPLY: starting
0506 11:39:17.595019 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:17.595065 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:17.595091 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:17.595171 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:17.595200 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:17.595204 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:17.595315 (+ 111us) log.cc:842] Serialized 22 byte log entry
0506 11:39:17.595400 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:17.595417 (+ 17us) write_op.cc:454] Released schema lock
0506 11:39:17.595423 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":424,"num_ops":1,"prepare.queue_time_us":159242,"prepare.run_cpu_time_us":697,"prepare.run_wall_time_us":364760,"replication_time_us":695833,"thread_start_us":287,"threads_started":1}]]}
I20240506 11:39:17.887207 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:17.887380 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:17.888137 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:17.888537 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:17.888622 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:17.889261 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:18.070663 13244 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:18.070842 13244 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:18.071609 13244 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:18.072286 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:18.072320 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:18.072904 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:18.073421 13244 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:18.083230 13289 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=10, attempt_no=0}) took 1137 ms. Trace:
I20240506 11:39:18.083357 13289 rpcz_store.cc:276] 0506 11:39:16.945470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:16.945532 (+ 62us) service_pool.cc:225] Handling call
0506 11:39:18.083162 (+1137630us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:16.946099 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:16.946199 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:16.946203 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:16.946205 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:16.946285 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:16.946296 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:16.946300 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:16.946339 (+ 39us) tablet.cc:699] Row locks acquired
0506 11:39:16.946341 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:16.946370 (+ 29us) write_op.cc:270] Start()
0506 11:39:17.259506 (+313136us) write_op.cc:276] Timestamp: P: 1714995556946323 usec, L: 0
0506 11:39:17.259510 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:17.259800 (+ 290us) log.cc:842] Serialized 354 byte log entry
0506 11:39:18.082543 (+822743us) op_driver.cc:444] REPLICATION: finished
0506 11:39:18.082630 (+ 87us) write_op.cc:301] APPLY: starting
0506 11:39:18.082647 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:18.082688 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:18.082721 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:18.082792 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:18.082826 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:18.082837 (+ 11us) write_op.cc:312] APPLY: finished
0506 11:39:18.082919 (+ 82us) log.cc:842] Serialized 22 byte log entry
0506 11:39:18.083052 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:18.083069 (+ 17us) write_op.cc:454] Released schema lock
0506 11:39:18.083075 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":714,"prepare.run_wall_time_us":313779,"replication_time_us":822993}]]}
W20240506 11:39:18.085276 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.989s user 0.001s sys 0.000s
W20240506 11:39:18.085847 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.989s user 0.000s sys 0.000s
I20240506 11:39:18.144975 13244 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:18.145176 13244 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:18.145876 13244 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:18.146258 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:18.146286 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:18.146853 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:18.147115 13244 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:18.189006 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:18.777310 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:18.777503 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:18.778364 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:18.778995 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:18.779057 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:18.779688 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:19.098868 13244 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:19.099092 13244 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:19.099761 13244 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:19.100343 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:19.100370 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:19.100982 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:19.107704 13302 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=12, attempt_no=0}) took 1021 ms. Trace:
I20240506 11:39:19.107872 13302 rpcz_store.cc:276] 0506 11:39:18.085784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:18.085861 (+ 77us) service_pool.cc:225] Handling call
0506 11:39:19.107649 (+1021788us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:18.133689 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:18.133815 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:18.133824 (+ 9us) write_op.cc:435] Acquired schema lock
0506 11:39:18.133829 (+ 5us) tablet.cc:660] Decoding operations
0506 11:39:18.133926 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:18.133936 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:18.133939 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:18.133985 (+ 46us) tablet.cc:699] Row locks acquired
0506 11:39:18.133986 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:18.134030 (+ 44us) write_op.cc:270] Start()
0506 11:39:18.273166 (+139136us) write_op.cc:276] Timestamp: P: 1714995558133962 usec, L: 0
0506 11:39:18.273170 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:18.273446 (+ 276us) log.cc:842] Serialized 354 byte log entry
0506 11:39:19.106609 (+833163us) op_driver.cc:444] REPLICATION: finished
0506 11:39:19.107080 (+ 471us) write_op.cc:301] APPLY: starting
0506 11:39:19.107095 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:19.107131 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:19.107200 (+ 69us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:19.107271 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:19.107314 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:19.107321 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:39:19.107432 (+ 111us) log.cc:842] Serialized 22 byte log entry
0506 11:39:19.107543 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:19.107558 (+ 15us) write_op.cc:454] Released schema lock
0506 11:39:19.107565 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":403,"num_ops":1,"prepare.queue_time_us":47407,"prepare.run_cpu_time_us":767,"prepare.run_wall_time_us":139837,"replication_time_us":833410,"thread_start_us":291,"threads_started":1}]]}
W20240506 11:39:19.109726 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.874s user 0.001s sys 0.000s
W20240506 11:39:19.109829 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.873s user 0.001s sys 0.000s
I20240506 11:39:19.411412 13244 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:19.444012 13302 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=13, attempt_no=0}) took 1294 ms. Trace:
I20240506 11:39:19.444161 13302 rpcz_store.cc:276] 0506 11:39:18.150004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:18.150065 (+ 61us) service_pool.cc:225] Handling call
0506 11:39:19.443952 (+1293887us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:18.273559 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:18.273673 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:18.273680 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:39:18.273684 (+ 4us) tablet.cc:660] Decoding operations
0506 11:39:18.273782 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:18.273794 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:18.273801 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:18.273848 (+ 47us) tablet.cc:699] Row locks acquired
0506 11:39:18.273850 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:18.273880 (+ 30us) write_op.cc:270] Start()
0506 11:39:18.553046 (+279166us) write_op.cc:276] Timestamp: P: 1714995558273846 usec, L: 0
0506 11:39:18.553051 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:18.553324 (+ 273us) log.cc:842] Serialized 354 byte log entry
0506 11:39:19.443336 (+890012us) op_driver.cc:444] REPLICATION: finished
0506 11:39:19.443433 (+ 97us) write_op.cc:301] APPLY: starting
0506 11:39:19.443472 (+ 39us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:19.443516 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:19.443550 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:19.443623 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:19.443658 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:19.443676 (+ 18us) write_op.cc:312] APPLY: finished
0506 11:39:19.443746 (+ 70us) log.cc:842] Serialized 22 byte log entry
0506 11:39:19.443829 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:19.443848 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:19.443855 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31,"num_ops":1,"prepare.queue_time_us":123009,"prepare.run_cpu_time_us":766,"prepare.run_wall_time_us":279847,"replication_time_us":890259}]]}
I20240506 11:39:19.508399 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:19.757349 13244 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:19.757588 13244 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:19.758327 13244 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:19.758776 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:19.758832 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:19.759547 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:19.759847 13244 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:19.765341 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:19.765534 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:19.766208 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:19.766914 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:19.766983 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:19.767818 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:19.768069 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 11:39:19.782644 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.522s user 0.001s sys 0.000s
W20240506 11:39:19.782730 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.522s user 0.001s sys 0.000s
I20240506 11:39:20.255671 13244 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:20.255846 13244 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:20.256583 13244 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:20.257036 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:20.257051 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:20.257652 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:20.258026 13244 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:20.682796 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:20.683032 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:20.683847 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:20.684168 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:20.684288 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:20.685442 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:20.685863 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:20.996088 13312 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:20.996292 13312 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:20.997200 13312 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:20.997666 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:20.997709 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:20.998485 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:20.998865 13312 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:21.014192 13315 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=16, attempt_no=0}) took 1230 ms. Trace:
I20240506 11:39:21.014398 13315 rpcz_store.cc:276] 0506 11:39:19.783482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:19.783533 (+ 51us) service_pool.cc:225] Handling call
0506 11:39:21.014125 (+1230592us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:19.987718 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:19.987888 (+ 170us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:19.987894 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:19.987896 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:19.987991 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:19.988004 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:19.988010 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:19.988068 (+ 58us) tablet.cc:699] Row locks acquired
0506 11:39:19.988072 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:39:19.988121 (+ 49us) write_op.cc:270] Start()
0506 11:39:20.753240 (+765119us) write_op.cc:276] Timestamp: P: 1714995559988066 usec, L: 0
0506 11:39:20.753243 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 11:39:20.753522 (+ 279us) log.cc:842] Serialized 354 byte log entry
0506 11:39:21.013135 (+259613us) op_driver.cc:444] REPLICATION: finished
0506 11:39:21.013549 (+ 414us) write_op.cc:301] APPLY: starting
0506 11:39:21.013565 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:21.013608 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:21.013631 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:21.013703 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:21.013731 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:21.013736 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:21.013842 (+ 106us) log.cc:842] Serialized 22 byte log entry
0506 11:39:21.014005 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:21.014023 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:21.014030 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":347,"mutex_wait_us":26,"num_ops":1,"prepare.queue_time_us":203739,"prepare.run_cpu_time_us":850,"prepare.run_wall_time_us":765905,"raft.queue_time_us":27,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":462,"replication_time_us":259862,"spinlock_wait_cycles":20096,"thread_start_us":230,"threads_started":1}]]}
W20240506 11:39:21.016666 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.568s user 0.001s sys 0.000s
W20240506 11:39:21.016736 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.568s user 0.001s sys 0.000s
I20240506 11:39:21.068444 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:21.068631 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:21.069362 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:21.069757 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:21.069952 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:21.071478 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:21.071743 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:21.077329 13312 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:21.077478 13312 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:21.078078 13312 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:21.078374 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:21.079141 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:21.079578 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:21.080019 13312 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:21.109748 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:21.109913 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:21.110622 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:21.111083 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:21.111306 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:21.111828 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:21.112084 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:21.661088 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:21.661310 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:21.662005 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:21.662428 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:21.662570 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:21.663190 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:21.693454 13319 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=17, attempt_no=0}) took 1396 ms. Trace:
I20240506 11:39:21.693579 13319 rpcz_store.cc:276] 0506 11:39:20.297282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:20.297337 (+ 55us) service_pool.cc:225] Handling call
0506 11:39:21.693402 (+1396065us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:20.753677 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:20.753808 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:20.753814 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:20.753815 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:20.753893 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:20.753903 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:20.753906 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:20.753951 (+ 45us) tablet.cc:699] Row locks acquired
0506 11:39:20.753953 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:20.753990 (+ 37us) write_op.cc:270] Start()
0506 11:39:21.599153 (+845163us) write_op.cc:276] Timestamp: P: 1714995560753923 usec, L: 0
0506 11:39:21.599158 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:21.599440 (+ 282us) log.cc:842] Serialized 354 byte log entry
0506 11:39:21.692312 (+ 92872us) op_driver.cc:444] REPLICATION: finished
0506 11:39:21.692745 (+ 433us) write_op.cc:301] APPLY: starting
0506 11:39:21.692765 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:21.692802 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:21.692847 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:21.692926 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:21.692955 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:21.692963 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:39:21.693088 (+ 125us) log.cc:842] Serialized 22 byte log entry
0506 11:39:21.693228 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:21.693248 (+ 20us) write_op.cc:454] Released schema lock
0506 11:39:21.693256 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":346,"num_ops":1,"prepare.queue_time_us":455981,"prepare.run_cpu_time_us":761,"prepare.run_wall_time_us":845850,"raft.queue_time_us":42,"raft.run_cpu_time_us":23,"raft.run_wall_time_us":23,"replication_time_us":93105,"spinlock_wait_cycles":28544,"thread_start_us":236,"threads_started":1}]]}
W20240506 11:39:21.695578 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.528s user 0.001s sys 0.000s
W20240506 11:39:21.695623 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.528s user 0.001s sys 0.000s
I20240506 11:39:21.763371 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:21.763583 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:21.764364 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:21.764869 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:21.765242 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:21.765796 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:22.033375 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:22.041009 13319 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=18, attempt_no=0}) took 1024 ms. Trace:
I20240506 11:39:22.041124 13319 rpcz_store.cc:276] 0506 11:39:21.016625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:21.016675 (+ 50us) service_pool.cc:225] Handling call
0506 11:39:22.040943 (+1024268us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:21.599562 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:21.599704 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:21.599710 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:21.599713 (+ 3us) tablet.cc:660] Decoding operations
0506 11:39:21.599824 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:21.599835 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:21.599842 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:21.599904 (+ 62us) tablet.cc:699] Row locks acquired
0506 11:39:21.599906 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:21.599944 (+ 38us) write_op.cc:270] Start()
0506 11:39:21.619076 (+ 19132us) write_op.cc:276] Timestamp: P: 1714995561599902 usec, L: 0
0506 11:39:21.619080 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:21.619324 (+ 244us) log.cc:842] Serialized 354 byte log entry
0506 11:39:22.040369 (+421045us) op_driver.cc:444] REPLICATION: finished
0506 11:39:22.040461 (+ 92us) write_op.cc:301] APPLY: starting
0506 11:39:22.040479 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:22.040521 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:22.040556 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:22.040611 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:22.040653 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:22.040658 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:22.040742 (+ 84us) log.cc:842] Serialized 22 byte log entry
0506 11:39:22.040813 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:22.040831 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:22.040838 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"num_ops":1,"prepare.queue_time_us":582434,"prepare.run_cpu_time_us":765,"prepare.run_wall_time_us":19828,"replication_time_us":421263}]]}
I20240506 11:39:22.185613 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:22.733798 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:22.733999 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:22.734719 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:22.735193 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:22.735221 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:22.735805 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:22.750594 13327 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=19, attempt_no=0}) took 1054 ms. Trace:
I20240506 11:39:22.750703 13327 rpcz_store.cc:276] 0506 11:39:21.695635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:21.695695 (+ 60us) service_pool.cc:225] Handling call
0506 11:39:22.750543 (+1054848us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:21.696144 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:21.696228 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:21.696232 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:21.696233 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:21.696302 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:21.696310 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:21.696313 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:21.696346 (+ 33us) tablet.cc:699] Row locks acquired
0506 11:39:21.696347 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:21.696375 (+ 28us) write_op.cc:270] Start()
0506 11:39:22.362529 (+666154us) write_op.cc:276] Timestamp: P: 1714995561696325 usec, L: 0
0506 11:39:22.362534 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:22.362819 (+ 285us) log.cc:842] Serialized 354 byte log entry
0506 11:39:22.749711 (+386892us) op_driver.cc:444] REPLICATION: finished
0506 11:39:22.750081 (+ 370us) write_op.cc:301] APPLY: starting
0506 11:39:22.750095 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:22.750126 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:22.750167 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:22.750223 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:22.750245 (+ 22us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:22.750249 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:22.750308 (+ 59us) log.cc:842] Serialized 22 byte log entry
0506 11:39:22.750410 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:22.750428 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:22.750435 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":295,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":705,"prepare.run_wall_time_us":666781,"raft.queue_time_us":109,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":222,"replication_time_us":387150,"thread_start_us":198,"threads_started":1}]]}
W20240506 11:39:22.752485 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.906s user 0.001s sys 0.000s
W20240506 11:39:22.752593 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.906s user 0.001s sys 0.000s
I20240506 11:39:22.977643 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:22.977906 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:22.978694 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:22.979270 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:22.979326 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:22.980053 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:23.250242 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:23.294471 13333 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=20, attempt_no=0}) took 1251 ms. Trace:
I20240506 11:39:23.294605 13333 rpcz_store.cc:276] 0506 11:39:22.042825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:22.042883 (+ 58us) service_pool.cc:225] Handling call
0506 11:39:23.294406 (+1251523us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:22.362980 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:22.363104 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:22.363110 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:22.363112 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:22.363208 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:22.363234 (+ 26us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:22.363239 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:22.363297 (+ 58us) tablet.cc:699] Row locks acquired
0506 11:39:22.363299 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:22.363361 (+ 62us) write_op.cc:270] Start()
0506 11:39:22.615517 (+252156us) write_op.cc:276] Timestamp: P: 1714995562363325 usec, L: 0
0506 11:39:22.615522 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:22.615754 (+ 232us) log.cc:842] Serialized 354 byte log entry
0506 11:39:23.293443 (+677689us) op_driver.cc:444] REPLICATION: finished
0506 11:39:23.293906 (+ 463us) write_op.cc:301] APPLY: starting
0506 11:39:23.293928 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:23.293975 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:23.294005 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:23.294082 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:23.294109 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:23.294113 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:23.294220 (+ 107us) log.cc:842] Serialized 22 byte log entry
0506 11:39:23.294294 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:23.294312 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:23.294317 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":354,"num_ops":1,"prepare.queue_time_us":319664,"prepare.run_cpu_time_us":769,"prepare.run_wall_time_us":252861,"replication_time_us":677880,"thread_start_us":244,"threads_started":1}]]}
I20240506 11:39:23.410166 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:23.410365 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:23.411047 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:23.411522 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:23.411497 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:23.412189 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:23.412529 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 11:39:23.458849 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.556s user 0.001s sys 0.000s
W20240506 11:39:23.459439 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.556s user 0.001s sys 0.000s
I20240506 11:39:23.591544 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:23.742002 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:23.742218 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:23.743062 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:23.743468 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:23.743499 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:23.744153 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:23.744459 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:24.050909 13245 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:24.051134 13245 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:24.051775 13245 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:24.052304 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:24.052563 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:24.053136 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:24.053622 13245 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:24.515486 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:24.515664 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:24.516489 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:24.516923 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:24.517402 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:24.517939 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:24.518405 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:25.481325 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:25.481515 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:25.482318 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:25.482808 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:25.482862 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:25.483498 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:25.483785 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:25.508589 13347 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=23, attempt_no=0}) took 2050 ms. Trace:
I20240506 11:39:25.508764 13347 rpcz_store.cc:276] 0506 11:39:23.458492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:23.458554 (+ 62us) service_pool.cc:225] Handling call
0506 11:39:25.508519 (+2049965us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:23.889597 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:23.889718 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:23.889724 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:23.889726 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:23.889831 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:23.889841 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:23.889845 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:23.889899 (+ 54us) tablet.cc:699] Row locks acquired
0506 11:39:23.889901 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:23.889941 (+ 40us) write_op.cc:270] Start()
0506 11:39:24.635069 (+745128us) write_op.cc:276] Timestamp: P: 1714995563889853 usec, L: 0
0506 11:39:24.635073 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:24.635304 (+ 231us) log.cc:842] Serialized 354 byte log entry
0506 11:39:25.507390 (+872086us) op_driver.cc:444] REPLICATION: finished
0506 11:39:25.507892 (+ 502us) write_op.cc:301] APPLY: starting
0506 11:39:25.507908 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:25.507949 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:25.507978 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:25.508095 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:25.508121 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:25.508126 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:25.508232 (+ 106us) log.cc:842] Serialized 22 byte log entry
0506 11:39:25.508357 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:25.508380 (+ 23us) write_op.cc:454] Released schema lock
0506 11:39:25.508388 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":413,"num_ops":1,"prepare.queue_time_us":430639,"prepare.run_cpu_time_us":715,"prepare.run_wall_time_us":745770,"replication_time_us":872268,"thread_start_us":282,"threads_started":1}]]}
I20240506 11:39:25.510190 12846 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 55) took 1288 ms. Trace:
I20240506 11:39:25.510216 12844 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 56) took 1287 ms. Trace:
I20240506 11:39:25.510309 12846 rpcz_store.cc:276] 0506 11:39:24.222011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:24.222074 (+ 63us) service_pool.cc:225] Handling call
0506 11:39:24.222276 (+ 202us) tablet_service.cc:2780] Created scanner 92162e5d1132401f9d487671ead931bf for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 80d980ac63d243cf8286aff2b009c2d0
0506 11:39:24.223227 (+ 951us) tablet_service.cc:2920] Creating iterator
0506 11:39:24.223266 (+ 39us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:24.635766 (+412500us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:25.508403 (+872637us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1285118 microseconds
0506 11:39:25.508558 (+ 155us) tablet_service.cc:2945] Iterator created
0506 11:39:25.509169 (+ 611us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:25.509200 (+ 31us) tablet_service.cc:3010] has_more: true
0506 11:39:25.509237 (+ 37us) tablet_service.cc:3027] Continuing scan request
0506 11:39:25.509310 (+ 73us) tablet_service.cc:3091] Found scanner 92162e5d1132401f9d487671ead931bf for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 80d980ac63d243cf8286aff2b009c2d0
0506 11:39:25.510107 (+ 797us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":768}
I20240506 11:39:25.510344 12844 rpcz_store.cc:276] 0506 11:39:24.222221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:24.222299 (+ 78us) service_pool.cc:225] Handling call
0506 11:39:24.222449 (+ 150us) tablet_service.cc:2780] Created scanner e5f69844f5dd4302ad74c0bddac4d320 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 3872b578c7ff4449b016f9e718fc81b8
0506 11:39:24.223410 (+ 961us) tablet_service.cc:2920] Creating iterator
0506 11:39:24.223444 (+ 34us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:24.635824 (+412380us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:25.508398 (+872574us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1284925 microseconds
0506 11:39:25.508557 (+ 159us) tablet_service.cc:2945] Iterator created
0506 11:39:25.509103 (+ 546us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:25.509139 (+ 36us) tablet_service.cc:3010] has_more: true
0506 11:39:25.509174 (+ 35us) tablet_service.cc:3027] Continuing scan request
0506 11:39:25.509238 (+ 64us) tablet_service.cc:3091] Found scanner e5f69844f5dd4302ad74c0bddac4d320 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 3872b578c7ff4449b016f9e718fc81b8
0506 11:39:25.510135 (+ 897us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":27136}
W20240506 11:39:25.511030 13240 scanner-internal.cc:458] Time spent opening tablet: real 1.290s user 0.001s sys 0.000s
W20240506 11:39:25.511077 13239 scanner-internal.cc:458] Time spent opening tablet: real 1.290s user 0.000s sys 0.001s
I20240506 11:39:25.628962 13341 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:25.629168 13341 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:25.630066 13341 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:25.630559 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:25.630764 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:25.631405 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:25.811941 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:25.812189 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:25.813073 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:25.813647 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:25.813721 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:25.814258 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:25.814674 13347 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=24, attempt_no=0}) took 1743 ms. Trace:
I20240506 11:39:25.814865 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:25.814776 13347 rpcz_store.cc:276] 0506 11:39:24.071423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:24.071489 (+ 66us) service_pool.cc:225] Handling call
0506 11:39:25.814621 (+1743132us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:24.635403 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:24.635502 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:24.635506 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:24.635507 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:24.635587 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:24.635602 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:24.635608 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:24.635662 (+ 54us) tablet.cc:699] Row locks acquired
0506 11:39:24.635666 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:39:24.635710 (+ 44us) write_op.cc:270] Start()
0506 11:39:25.359855 (+724145us) write_op.cc:276] Timestamp: P: 1714995564635690 usec, L: 0
0506 11:39:25.359859 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:25.360079 (+ 220us) log.cc:842] Serialized 354 byte log entry
0506 11:39:25.814053 (+453974us) op_driver.cc:444] REPLICATION: finished
0506 11:39:25.814177 (+ 124us) write_op.cc:301] APPLY: starting
0506 11:39:25.814194 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:25.814227 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:25.814253 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:25.814319 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:25.814350 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:25.814360 (+ 10us) write_op.cc:312] APPLY: finished
0506 11:39:25.814440 (+ 80us) log.cc:842] Serialized 22 byte log entry
0506 11:39:25.814504 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:25.814534 (+ 30us) write_op.cc:454] Released schema lock
0506 11:39:25.814542 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":69,"num_ops":1,"prepare.queue_time_us":563493,"prepare.run_cpu_time_us":683,"prepare.run_wall_time_us":724744,"replication_time_us":454162}]]}
I20240506 11:39:26.059307 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:26.059535 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:26.060245 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:26.061002 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:26.061596 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:26.062103 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:26.062408 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:26.496634 13341 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:26.502367 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:26.502568 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:26.503269 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:26.503772 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:26.503770 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:26.504312 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:26.504575 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:26.548048 13356 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=25, attempt_no=0}) took 1037 ms. Trace:
I20240506 11:39:26.548178 13356 rpcz_store.cc:276] 0506 11:39:25.511070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:25.511128 (+ 58us) service_pool.cc:225] Handling call
0506 11:39:26.547991 (+1036863us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:25.511631 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:25.511703 (+ 72us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:25.511707 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:25.511708 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:25.511770 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:25.511779 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:25.511782 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:25.511816 (+ 34us) tablet.cc:699] Row locks acquired
0506 11:39:25.511817 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:25.511845 (+ 28us) write_op.cc:270] Start()
0506 11:39:26.222991 (+711146us) write_op.cc:276] Timestamp: P: 1714995565511832 usec, L: 0
0506 11:39:26.222996 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:26.223277 (+ 281us) log.cc:842] Serialized 354 byte log entry
0506 11:39:26.546870 (+323593us) op_driver.cc:444] REPLICATION: finished
0506 11:39:26.547362 (+ 492us) write_op.cc:301] APPLY: starting
0506 11:39:26.547384 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:26.547434 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:26.547467 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:26.547588 (+ 121us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:26.547628 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:26.547635 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:39:26.547730 (+ 95us) log.cc:842] Serialized 22 byte log entry
0506 11:39:26.547863 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:26.547884 (+ 21us) write_op.cc:454] Released schema lock
0506 11:39:26.547891 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":412,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":637,"prepare.run_wall_time_us":711714,"replication_time_us":323807,"thread_start_us":294,"threads_started":1}]]}
W20240506 11:39:26.550536 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.889s user 0.000s sys 0.001s
W20240506 11:39:26.550577 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.889s user 0.001s sys 0.000s
I20240506 11:39:27.059959 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:27.060146 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:27.061087 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:27.061496 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:27.061496 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:27.062109 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:27.062458 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:27.101730 13361 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=26, attempt_no=0}) took 1285 ms. Trace:
I20240506 11:39:27.101861 13361 rpcz_store.cc:276] 0506 11:39:25.816411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:25.816467 (+ 56us) service_pool.cc:225] Handling call
0506 11:39:27.101686 (+1285219us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:26.223383 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:26.223507 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:26.223514 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:39:26.223519 (+ 5us) tablet.cc:660] Decoding operations
0506 11:39:26.223628 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:26.223639 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:26.223644 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:26.223702 (+ 58us) tablet.cc:699] Row locks acquired
0506 11:39:26.223704 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:26.223742 (+ 38us) write_op.cc:270] Start()
0506 11:39:26.599926 (+376184us) write_op.cc:276] Timestamp: P: 1714995566223721 usec, L: 0
0506 11:39:26.599930 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:26.600133 (+ 203us) log.cc:842] Serialized 354 byte log entry
0506 11:39:27.100974 (+500841us) op_driver.cc:444] REPLICATION: finished
0506 11:39:27.101299 (+ 325us) write_op.cc:301] APPLY: starting
0506 11:39:27.101313 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:27.101346 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:27.101366 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:27.101432 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:27.101454 (+ 22us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:27.101459 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:27.101538 (+ 79us) log.cc:842] Serialized 22 byte log entry
0506 11:39:27.101596 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:27.101612 (+ 16us) write_op.cc:454] Released schema lock
0506 11:39:27.101617 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":255,"mutex_wait_us":44,"num_ops":1,"prepare.queue_time_us":406493,"prepare.run_cpu_time_us":813,"prepare.run_wall_time_us":376899,"raft.queue_time_us":245,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":501007,"thread_start_us":177,"threads_started":1}]]}
I20240506 11:39:27.281594 13341 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:27.281817 13341 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:27.282738 13341 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:27.283169 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:27.283288 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:27.284039 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:27.930497 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:27.930691 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:27.931491 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:27.932205 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:27.932476 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:27.933030 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:27.933296 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:27.962340 13368 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=27, attempt_no=0}) took 1411 ms. Trace:
I20240506 11:39:27.962586 13368 rpcz_store.cc:276] 0506 11:39:26.551015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:26.551065 (+ 50us) service_pool.cc:225] Handling call
0506 11:39:27.962269 (+1411204us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:26.600311 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:26.600404 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:26.600408 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:26.600410 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:26.600510 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:26.600525 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:26.600530 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:26.600585 (+ 55us) tablet.cc:699] Row locks acquired
0506 11:39:26.600588 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:39:26.600629 (+ 41us) write_op.cc:270] Start()
0506 11:39:27.097777 (+497148us) write_op.cc:276] Timestamp: P: 1714995566600608 usec, L: 0
0506 11:39:27.097782 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:27.098086 (+ 304us) log.cc:842] Serialized 354 byte log entry
0506 11:39:27.961069 (+862983us) op_driver.cc:444] REPLICATION: finished
0506 11:39:27.961542 (+ 473us) write_op.cc:301] APPLY: starting
0506 11:39:27.961560 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:27.961594 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:27.961643 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:27.961719 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:27.961760 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:27.961769 (+ 9us) write_op.cc:312] APPLY: finished
0506 11:39:27.961878 (+ 109us) log.cc:842] Serialized 22 byte log entry
0506 11:39:27.962066 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:27.962088 (+ 22us) write_op.cc:454] Released schema lock
0506 11:39:27.962139 (+ 51us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":395,"num_ops":1,"prepare.queue_time_us":48862,"prepare.run_cpu_time_us":810,"prepare.run_wall_time_us":497877,"raft.queue_time_us":83,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":229,"replication_time_us":863211,"spinlock_wait_cycles":34176,"thread_start_us":252,"threads_started":1}]]}
I20240506 11:39:27.964152 12846 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 63) took 1262 ms. Trace:
I20240506 11:39:27.964231 12844 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 62) took 1262 ms. Trace:
I20240506 11:39:27.964366 12844 rpcz_store.cc:276] 0506 11:39:26.701893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:26.701968 (+ 75us) service_pool.cc:225] Handling call
0506 11:39:26.702182 (+ 214us) tablet_service.cc:2780] Created scanner 52ae74545dd54954a7e1ca899c0c5f9c for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is c48d235a354e4ff185705eeae980eb06
0506 11:39:26.702981 (+ 799us) tablet_service.cc:2920] Creating iterator
0506 11:39:26.703033 (+ 52us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:27.097935 (+394902us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:27.962054 (+864119us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1258997 microseconds
0506 11:39:27.962218 (+ 164us) tablet_service.cc:2945] Iterator created
0506 11:39:27.962884 (+ 666us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:27.962918 (+ 34us) tablet_service.cc:3010] has_more: true
0506 11:39:27.963000 (+ 82us) tablet_service.cc:3027] Continuing scan request
0506 11:39:27.963111 (+ 111us) tablet_service.cc:3091] Found scanner 52ae74545dd54954a7e1ca899c0c5f9c for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is c48d235a354e4ff185705eeae980eb06
0506 11:39:27.964163 (+ 1052us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":28928}
I20240506 11:39:27.964265 12846 rpcz_store.cc:276] 0506 11:39:26.701999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:26.702075 (+ 76us) service_pool.cc:225] Handling call
0506 11:39:26.702229 (+ 154us) tablet_service.cc:2780] Created scanner 22b3dae07f0b4b44ba32bb0895d4f380 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 780403833d8943e7b4e61d57386071d6
0506 11:39:26.702986 (+ 757us) tablet_service.cc:2920] Creating iterator
0506 11:39:26.703036 (+ 50us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:27.097940 (+394904us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:27.962161 (+864221us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1259110 microseconds
0506 11:39:27.962250 (+ 89us) tablet_service.cc:2945] Iterator created
0506 11:39:27.962858 (+ 608us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:27.962896 (+ 38us) tablet_service.cc:3010] has_more: true
0506 11:39:27.962979 (+ 83us) tablet_service.cc:3027] Continuing scan request
0506 11:39:27.963067 (+ 88us) tablet_service.cc:3091] Found scanner 22b3dae07f0b4b44ba32bb0895d4f380 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 780403833d8943e7b4e61d57386071d6
0506 11:39:27.964094 (+ 1027us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":25856}
W20240506 11:39:27.965399 13240 scanner-internal.cc:458] Time spent opening tablet: real 1.264s user 0.001s sys 0.000s
W20240506 11:39:27.965440 13239 scanner-internal.cc:458] Time spent opening tablet: real 1.264s user 0.000s sys 0.001s
I20240506 11:39:28.023778 13341 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:28.595072 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:28.595259 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:28.595968 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:28.596436 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:28.596452 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:28.597231 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:28.640084 13341 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:28.640267 13341 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:28.640913 13341 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:28.641480 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:28.641547 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:28.642241 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:28.642671 13341 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 11:39:28.651011 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.535s user 0.001s sys 0.000s
W20240506 11:39:28.651158 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.535s user 0.000s sys 0.001s
I20240506 11:39:28.680052 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:29.411595 13316 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:29.411810 13316 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:29.413110 13316 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:29.413599 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:29.413877 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:29.414525 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:29.457163 13381 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=30, attempt_no=0}) took 1391 ms. Trace:
I20240506 11:39:29.457362 13381 rpcz_store.cc:276] 0506 11:39:28.065765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:28.065823 (+ 58us) service_pool.cc:225] Handling call
0506 11:39:29.457074 (+1391251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:28.247470 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:28.247610 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:28.247618 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:39:28.247621 (+ 3us) tablet.cc:660] Decoding operations
0506 11:39:28.247701 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:28.247711 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:28.247714 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:28.247762 (+ 48us) tablet.cc:699] Row locks acquired
0506 11:39:28.247764 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:28.247807 (+ 43us) write_op.cc:270] Start()
0506 11:39:29.096923 (+849116us) write_op.cc:276] Timestamp: P: 1714995568247777 usec, L: 0
0506 11:39:29.096927 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:29.097208 (+ 281us) log.cc:842] Serialized 354 byte log entry
0506 11:39:29.456100 (+358892us) op_driver.cc:444] REPLICATION: finished
0506 11:39:29.456530 (+ 430us) write_op.cc:301] APPLY: starting
0506 11:39:29.456547 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:29.456580 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:29.456610 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:29.456687 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:29.456712 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:29.456716 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:29.456793 (+ 77us) log.cc:842] Serialized 22 byte log entry
0506 11:39:29.456936 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:29.456956 (+ 20us) write_op.cc:454] Released schema lock
0506 11:39:29.456963 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":363,"num_ops":1,"prepare.queue_time_us":181047,"prepare.run_cpu_time_us":785,"prepare.run_wall_time_us":849838,"raft.queue_time_us":251,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":236,"replication_time_us":359137,"spinlock_wait_cycles":23936,"thread_start_us":251,"threads_started":1}]]}
W20240506 11:39:29.459771 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.658s user 0.001s sys 0.000s
W20240506 11:39:29.459802 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.658s user 0.000s sys 0.001s
I20240506 11:39:29.733274 13316 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:29.737932 13381 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=31, attempt_no=0}) took 1086 ms. Trace:
I20240506 11:39:29.738077 13381 rpcz_store.cc:276] 0506 11:39:28.651209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:28.651304 (+ 95us) service_pool.cc:225] Handling call
0506 11:39:29.737856 (+1086552us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:29.097341 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:29.097465 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:29.097471 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:29.097473 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:29.097585 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:29.097596 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:29.097603 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:29.097648 (+ 45us) tablet.cc:699] Row locks acquired
0506 11:39:29.097650 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:29.097684 (+ 34us) write_op.cc:270] Start()
0506 11:39:29.408855 (+311171us) write_op.cc:276] Timestamp: P: 1714995569097665 usec, L: 0
0506 11:39:29.408868 (+ 13us) op_driver.cc:331] REPLICATION: starting
0506 11:39:29.409146 (+ 278us) log.cc:842] Serialized 354 byte log entry
0506 11:39:29.737319 (+328173us) op_driver.cc:444] REPLICATION: finished
0506 11:39:29.737440 (+ 121us) write_op.cc:301] APPLY: starting
0506 11:39:29.737458 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:29.737494 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:29.737528 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:29.737588 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:29.737612 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:29.737617 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:29.737681 (+ 64us) log.cc:842] Serialized 22 byte log entry
0506 11:39:29.737751 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:29.737766 (+ 15us) write_op.cc:454] Released schema lock
0506 11:39:29.737772 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"num_ops":1,"prepare.queue_time_us":445608,"prepare.run_cpu_time_us":831,"prepare.run_wall_time_us":311903,"replication_time_us":328406}]]}
I20240506 11:39:30.067817 13341 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:30.067984 13341 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:30.068683 13341 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:30.069223 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:30.070226 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:30.070861 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:30.939163 13385 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:30.939400 13385 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:30.940196 13385 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:30.940728 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:30.940831 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:30.941468 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:30.941740 13385 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:30.963732 13391 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=32, attempt_no=0}) took 1503 ms. Trace:
I20240506 11:39:30.963927 13391 rpcz_store.cc:276] 0506 11:39:29.460168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:29.460242 (+ 74us) service_pool.cc:225] Handling call
0506 11:39:30.963658 (+1503416us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:29.460825 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:29.460909 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:29.460913 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:29.460914 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:29.460976 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:29.460984 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:29.460987 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:29.461020 (+ 33us) tablet.cc:699] Row locks acquired
0506 11:39:29.461021 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:29.461049 (+ 28us) write_op.cc:270] Start()
0506 11:39:30.427190 (+966141us) write_op.cc:276] Timestamp: P: 1714995569461025 usec, L: 0
0506 11:39:30.427196 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:39:30.427453 (+ 257us) log.cc:842] Serialized 354 byte log entry
0506 11:39:30.962528 (+535075us) op_driver.cc:444] REPLICATION: finished
0506 11:39:30.963034 (+ 506us) write_op.cc:301] APPLY: starting
0506 11:39:30.963049 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:30.963081 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:30.963106 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:30.963257 (+ 151us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:30.963283 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:30.963288 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:30.963395 (+ 107us) log.cc:842] Serialized 22 byte log entry
0506 11:39:30.963495 (+ 100us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:30.963516 (+ 21us) write_op.cc:454] Released schema lock
0506 11:39:30.963522 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":430,"num_ops":1,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":642,"prepare.run_wall_time_us":966699,"replication_time_us":535304,"thread_start_us":310,"threads_started":1}]]}
I20240506 11:39:30.965618 12846 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 73) took 1354 ms. Trace:
I20240506 11:39:30.965670 12844 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 74) took 1354 ms. Trace:
I20240506 11:39:30.965787 12844 rpcz_store.cc:276] 0506 11:39:29.611316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:29.611374 (+ 58us) service_pool.cc:225] Handling call
0506 11:39:29.611521 (+ 147us) tablet_service.cc:2780] Created scanner 3bb4a8ba80cf46bcbdb5b6f3f2faf728 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 3b5f69851a77400fbafeff19d8072e1e
0506 11:39:29.612276 (+ 755us) tablet_service.cc:2920] Creating iterator
0506 11:39:29.612330 (+ 54us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:30.427324 (+814994us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:30.963550 (+536226us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1351187 microseconds
0506 11:39:30.963658 (+ 108us) tablet_service.cc:2945] Iterator created
0506 11:39:30.964331 (+ 673us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:30.964363 (+ 32us) tablet_service.cc:3010] has_more: true
0506 11:39:30.964398 (+ 35us) tablet_service.cc:3027] Continuing scan request
0506 11:39:30.964473 (+ 75us) tablet_service.cc:3091] Found scanner 3bb4a8ba80cf46bcbdb5b6f3f2faf728 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 3b5f69851a77400fbafeff19d8072e1e
0506 11:39:30.965600 (+ 1127us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20240506 11:39:30.965736 12846 rpcz_store.cc:276] 0506 11:39:29.611175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:29.611256 (+ 81us) service_pool.cc:225] Handling call
0506 11:39:29.611527 (+ 271us) tablet_service.cc:2780] Created scanner b614ed4023df4ed0834835373d96c18b for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is ddca23670fc4472fbcf9a3f7db908d1c
0506 11:39:29.612588 (+ 1061us) tablet_service.cc:2920] Creating iterator
0506 11:39:29.612627 (+ 39us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:30.427326 (+814699us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:30.963557 (+536231us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1350914 microseconds
0506 11:39:30.963675 (+ 118us) tablet_service.cc:2945] Iterator created
0506 11:39:30.964279 (+ 604us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:30.964312 (+ 33us) tablet_service.cc:3010] has_more: true
0506 11:39:30.964345 (+ 33us) tablet_service.cc:3027] Continuing scan request
0506 11:39:30.964408 (+ 63us) tablet_service.cc:3091] Found scanner b614ed4023df4ed0834835373d96c18b for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is ddca23670fc4472fbcf9a3f7db908d1c
0506 11:39:30.965557 (+ 1149us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":32384}
W20240506 11:39:30.966550 13240 scanner-internal.cc:458] Time spent opening tablet: real 1.356s user 0.001s sys 0.000s
W20240506 11:39:30.966575 13239 scanner-internal.cc:458] Time spent opening tablet: real 1.356s user 0.000s sys 0.001s
I20240506 11:39:31.062516 13341 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:31.626724 13385 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:31.627048 13385 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:31.627821 13385 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:31.628245 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:31.628309 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:31.628911 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:31.629133 13385 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:31.645714 13398 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=33, attempt_no=0}) took 1905 ms. Trace:
I20240506 11:39:31.645927 13398 rpcz_store.cc:276] 0506 11:39:29.739744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:29.739812 (+ 68us) service_pool.cc:225] Handling call
0506 11:39:31.645642 (+1905830us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:30.427563 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:30.427687 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:30.427695 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:39:30.427698 (+ 3us) tablet.cc:660] Decoding operations
0506 11:39:30.427802 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:30.427812 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:30.427815 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:30.427857 (+ 42us) tablet.cc:699] Row locks acquired
0506 11:39:30.427858 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:30.427892 (+ 34us) write_op.cc:270] Start()
0506 11:39:31.119458 (+691566us) write_op.cc:276] Timestamp: P: 1714995570427866 usec, L: 0
0506 11:39:31.119462 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:31.119726 (+ 264us) log.cc:842] Serialized 354 byte log entry
0506 11:39:31.643602 (+523876us) op_driver.cc:444] REPLICATION: finished
0506 11:39:31.644053 (+ 451us) write_op.cc:301] APPLY: starting
0506 11:39:31.644072 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:31.644113 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:31.644135 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:31.644221 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:31.644247 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:31.644251 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:31.644348 (+ 97us) log.cc:842] Serialized 22 byte log entry
0506 11:39:31.645512 (+ 1164us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:31.645533 (+ 21us) write_op.cc:454] Released schema lock
0506 11:39:31.645541 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":377,"num_ops":1,"prepare.queue_time_us":687416,"prepare.run_cpu_time_us":735,"prepare.run_wall_time_us":692239,"replication_time_us":524096,"thread_start_us":237,"threads_started":1}]]}
W20240506 11:39:31.646780 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.529s user 0.000s sys 0.000s
W20240506 11:39:31.646857 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.530s user 0.001s sys 0.000s
I20240506 11:39:31.665360 13385 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:31.665593 13385 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:31.666328 13385 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:31.666713 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:31.666777 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:31.667452 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:31.667717 13385 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:31.879859 13341 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:31.880036 13341 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:31.880765 13341 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:31.881338 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:31.881894 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:31.882733 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:32.307408 13402 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:32.307648 13402 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:32.308496 13402 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:32.308977 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:32.308969 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:32.309615 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
W20240506 11:39:32.326362 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.529s user 0.001s sys 0.000s
W20240506 11:39:32.326418 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.529s user 0.001s sys 0.000s
I20240506 11:39:32.828181 13341 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:33.152662 13402 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:33.194398 13412 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=36, attempt_no=0}) took 1524 ms. Trace:
I20240506 11:39:33.194561 13412 rpcz_store.cc:276] 0506 11:39:31.670232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:31.670289 (+ 57us) service_pool.cc:225] Handling call
0506 11:39:33.194337 (+1524048us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:32.268972 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:32.269082 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:32.269087 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:32.269088 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:32.269217 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:32.269228 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:32.269233 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:32.269282 (+ 49us) tablet.cc:699] Row locks acquired
0506 11:39:32.269284 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:32.269319 (+ 35us) write_op.cc:270] Start()
0506 11:39:32.282471 (+ 13152us) write_op.cc:276] Timestamp: P: 1714995572269283 usec, L: 0
0506 11:39:32.282476 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:32.282660 (+ 184us) log.cc:842] Serialized 354 byte log entry
0506 11:39:33.193164 (+910504us) op_driver.cc:444] REPLICATION: finished
0506 11:39:33.193815 (+ 651us) write_op.cc:301] APPLY: starting
0506 11:39:33.193833 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:33.193867 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:33.193893 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:33.193971 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:33.193996 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:33.194001 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:33.194095 (+ 94us) log.cc:842] Serialized 22 byte log entry
0506 11:39:33.194178 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:33.194198 (+ 20us) write_op.cc:454] Released schema lock
0506 11:39:33.194206 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":582,"num_ops":1,"prepare.queue_time_us":598339,"prepare.run_cpu_time_us":680,"prepare.run_wall_time_us":13748,"replication_time_us":910611,"thread_start_us":467,"threads_started":1}]]}
I20240506 11:39:33.594041 13402 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:33.594254 13402 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:33.595081 13402 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:33.595628 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:33.595654 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:33.596328 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:33.597268 13412 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=37, attempt_no=0}) took 1271 ms. Trace:
I20240506 11:39:33.597389 13412 rpcz_store.cc:276] 0506 11:39:32.325932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:32.325993 (+ 61us) service_pool.cc:225] Handling call
0506 11:39:33.597179 (+1271186us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:32.326511 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:32.326598 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:32.326606 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:39:32.326610 (+ 4us) tablet.cc:660] Decoding operations
0506 11:39:32.326689 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:32.326697 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:32.326700 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:32.326731 (+ 31us) tablet.cc:699] Row locks acquired
0506 11:39:32.326732 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:32.326759 (+ 27us) write_op.cc:270] Start()
0506 11:39:32.725921 (+399162us) write_op.cc:276] Timestamp: P: 1714995572326723 usec, L: 0
0506 11:39:32.725926 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:32.726240 (+ 314us) log.cc:842] Serialized 354 byte log entry
0506 11:39:33.596510 (+870270us) op_driver.cc:444] REPLICATION: finished
0506 11:39:33.596607 (+ 97us) write_op.cc:301] APPLY: starting
0506 11:39:33.596622 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:33.596653 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:33.596675 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:33.596754 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:33.596783 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:33.596789 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:39:33.596874 (+ 85us) log.cc:842] Serialized 22 byte log entry
0506 11:39:33.597054 (+ 180us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:33.597070 (+ 16us) write_op.cc:454] Released schema lock
0506 11:39:33.597077 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":35,"num_ops":1,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":736,"prepare.run_wall_time_us":399815,"replication_time_us":870547}]]}
I20240506 11:39:33.599068 12846 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 85) took 1121 ms. Trace:
I20240506 11:39:33.599035 12845 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 84) took 1121 ms. Trace:
I20240506 11:39:33.599236 12845 rpcz_store.cc:276] 0506 11:39:32.477728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:32.477806 (+ 78us) service_pool.cc:225] Handling call
0506 11:39:32.478100 (+ 294us) tablet_service.cc:2780] Created scanner c07c7a7a8dfc457a9ddfb442d899101d for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 72b0f47858ef45adb9d5bb960f6e4247
0506 11:39:32.479080 (+ 980us) tablet_service.cc:2920] Creating iterator
0506 11:39:32.479125 (+ 45us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:32.726127 (+247002us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:33.597088 (+870961us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1117953 microseconds
0506 11:39:33.597199 (+ 111us) tablet_service.cc:2945] Iterator created
0506 11:39:33.597775 (+ 576us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:33.597810 (+ 35us) tablet_service.cc:3010] has_more: true
0506 11:39:33.597848 (+ 38us) tablet_service.cc:3027] Continuing scan request
0506 11:39:33.597924 (+ 76us) tablet_service.cc:3091] Found scanner c07c7a7a8dfc457a9ddfb442d899101d for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 72b0f47858ef45adb9d5bb960f6e4247
0506 11:39:33.598941 (+ 1017us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":116224}
I20240506 11:39:33.599222 12846 rpcz_store.cc:276] 0506 11:39:32.477826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:32.477884 (+ 58us) service_pool.cc:225] Handling call
0506 11:39:32.478100 (+ 216us) tablet_service.cc:2780] Created scanner 927fea1594864bacb7128ef8edf86300 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is af4c347ef6fc4f21a595b471728ceee0
0506 11:39:32.479086 (+ 986us) tablet_service.cc:2920] Creating iterator
0506 11:39:32.479125 (+ 39us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:32.726088 (+246963us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:33.597078 (+870990us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1117928 microseconds
0506 11:39:33.597199 (+ 121us) tablet_service.cc:2945] Iterator created
0506 11:39:33.597775 (+ 576us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:33.597814 (+ 39us) tablet_service.cc:3010] has_more: true
0506 11:39:33.597848 (+ 34us) tablet_service.cc:3027] Continuing scan request
0506 11:39:33.597924 (+ 76us) tablet_service.cc:3091] Found scanner 927fea1594864bacb7128ef8edf86300 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is af4c347ef6fc4f21a595b471728ceee0
0506 11:39:33.598941 (+ 1017us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":110336}
W20240506 11:39:33.600071 13240 scanner-internal.cc:458] Time spent opening tablet: real 1.123s user 0.001s sys 0.000s
W20240506 11:39:33.600081 13239 scanner-internal.cc:458] Time spent opening tablet: real 1.123s user 0.001s sys 0.000s
I20240506 11:39:34.325495 13402 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:34.354984 13418 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=38, attempt_no=0}) took 1158 ms. Trace:
I20240506 11:39:34.355134 13418 rpcz_store.cc:276] 0506 11:39:33.196341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:33.196401 (+ 60us) service_pool.cc:225] Handling call
0506 11:39:34.354896 (+1158495us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:33.196952 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:33.197063 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:33.197067 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:33.197068 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:33.197136 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:33.197144 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:33.197147 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:33.197184 (+ 37us) tablet.cc:699] Row locks acquired
0506 11:39:33.197185 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:33.197217 (+ 32us) write_op.cc:270] Start()
0506 11:39:33.370422 (+173205us) write_op.cc:276] Timestamp: P: 1714995573197196 usec, L: 0
0506 11:39:33.370428 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:39:33.370708 (+ 280us) log.cc:842] Serialized 354 byte log entry
0506 11:39:34.354018 (+983310us) op_driver.cc:444] REPLICATION: finished
0506 11:39:34.354419 (+ 401us) write_op.cc:301] APPLY: starting
0506 11:39:34.354435 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:34.354468 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:34.354492 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:34.354568 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:34.354593 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:34.354597 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:34.354670 (+ 73us) log.cc:842] Serialized 22 byte log entry
0506 11:39:34.354761 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:34.354780 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:34.354788 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":334,"num_ops":1,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":771,"prepare.run_wall_time_us":173841,"replication_time_us":983554,"thread_start_us":234,"threads_started":1}]]}
I20240506 11:39:34.423575 13402 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:34.423780 13402 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:34.424576 13402 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:34.425089 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:34.425104 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:34.425920 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:34.426162 13402 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 11:39:34.474200 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.723s user 0.001s sys 0.000s
W20240506 11:39:34.474870 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.724s user 0.001s sys 0.000s
I20240506 11:39:35.289019 13341 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:35.289322 13017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:39005 (request call id 2940) took 2459 ms. Trace:
I20240506 11:39:35.289242 13341 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:35.290024 13341 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:35.289469 13017 rpcz_store.cc:276] 0506 11:39:32.829499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:32.829584 (+ 85us) service_pool.cc:225] Handling call
0506 11:39:32.829771 (+ 187us) raft_consensus.cc:1457] Updating replica for 3 ops
0506 11:39:32.829906 (+ 135us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 11:39:32.829909 (+ 3us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0506 11:39:32.829991 (+ 82us) write_op.cc:270] Start()
0506 11:39:33.683135 (+853144us) write_op.cc:276] Timestamp: P: 1714995571648434 usec, L: 0
0506 11:39:33.683754 (+ 619us) write_op.cc:270] Start()
0506 11:39:34.556909 (+873155us) write_op.cc:276] Timestamp: P: 1714995572269283 usec, L: 0
0506 11:39:34.557415 (+ 506us) write_op.cc:270] Start()
0506 11:39:35.287582 (+730167us) write_op.cc:276] Timestamp: P: 1714995572326723 usec, L: 0
0506 11:39:35.288174 (+ 592us) log.cc:842] Serialized 1062 byte log entry
0506 11:39:35.288574 (+ 400us) raft_consensus.cc:1631] Marking committed up to 37
0506 11:39:35.288871 (+ 297us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:35.288932 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:35.289139 (+ 207us) raft_consensus.cc:1671] finished
0506 11:39:35.289143 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:35.289256 (+ 113us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:33.683664 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:33.683802 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:33.683808 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:33.683822 (+ 14us) tablet.cc:660] Decoding operations
0506 11:39:33.683913 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:33.683926 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:33.683930 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:33.683976 (+ 46us) tablet.cc:699] Row locks acquired
0506 11:39:33.683978 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:35.288612 (+1604634us) op_driver.cc:444] REPLICATION: finished
0506 11:39:35.288965 (+ 353us) write_op.cc:301] APPLY: starting
0506 11:39:35.288982 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:35.289028 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:35.289030 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:35.289123 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:35.289160 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:35.289195 (+ 35us) write_op.cc:312] APPLY: finished
0506 11:39:35.289309 (+ 114us) log.cc:842] Serialized 22 byte log entry
0506 11:39:35.289375 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:35.289390 (+ 15us) write_op.cc:454] Released schema lock
0506 11:39:35.289396 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:39:35.289481 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:35.289485 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:39:34.557351 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:34.557511 (+ 160us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:34.557517 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:34.557518 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:34.557609 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:34.557621 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:34.557624 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:34.557670 (+ 46us) tablet.cc:699] Row locks acquired
0506 11:39:34.557671 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0506 11:39:35.288064 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:35.288172 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:35.288178 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:35.288179 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:35.288287 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:35.288305 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:35.288309 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:35.288375 (+ 66us) tablet.cc:699] Row locks acquired
0506 11:39:35.288377 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":324,"threads_started":1,"wal-append.queue_time_us":388,"child_traces":[["op",{"apply.queue_time_us":276,"apply.run_cpu_time_us":581,"apply.run_wall_time_us":580,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":370,"prepare.run_cpu_time_us":399,"prepare.run_wall_time_us":417,"replication_time_us":2458607,"thread_start_us":423,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":314,"prepare.run_cpu_time_us":399,"prepare.run_wall_time_us":398,"thread_start_us":220,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":345,"prepare.run_cpu_time_us":380,"prepare.run_wall_time_us":381,"thread_start_us":235,"threads_started":1}]]}
I20240506 11:39:35.290562 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:35.290686 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:35.291227 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:35.291589 13341 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:35.517400 13420 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:35.517621 13420 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:35.518316 13420 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:35.518962 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:35.519006 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:35.519641 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:35.519968 13420 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:35.525523 13429 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=40, attempt_no=0}) took 1168 ms. Trace:
I20240506 11:39:35.525641 13429 rpcz_store.cc:276] 0506 11:39:34.356713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:34.356784 (+ 71us) service_pool.cc:225] Handling call
0506 11:39:35.525467 (+1168683us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:34.357248 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:34.357337 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:34.357341 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:34.357342 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:34.357428 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:34.357439 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:34.357443 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:34.357479 (+ 36us) tablet.cc:699] Row locks acquired
0506 11:39:34.357480 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:34.357511 (+ 31us) write_op.cc:270] Start()
0506 11:39:34.932666 (+575155us) write_op.cc:276] Timestamp: P: 1714995574357488 usec, L: 0
0506 11:39:34.932670 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:34.932966 (+ 296us) log.cc:842] Serialized 354 byte log entry
0506 11:39:35.524584 (+591618us) op_driver.cc:444] REPLICATION: finished
0506 11:39:35.525003 (+ 419us) write_op.cc:301] APPLY: starting
0506 11:39:35.525015 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:35.525043 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:35.525083 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:35.525150 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:35.525179 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:35.525185 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:39:35.525249 (+ 64us) log.cc:842] Serialized 22 byte log entry
0506 11:39:35.525354 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:35.525374 (+ 20us) write_op.cc:454] Released schema lock
0506 11:39:35.525380 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":352,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":716,"prepare.run_wall_time_us":575793,"replication_time_us":591888,"thread_start_us":238,"threads_started":1}]]}
W20240506 11:39:35.527468 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.903s user 0.001s sys 0.000s
W20240506 11:39:35.528789 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.903s user 0.000s sys 0.001s
I20240506 11:39:36.324301 13420 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:36.324568 13420 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:36.325243 13420 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:36.325721 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:36.325809 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:36.326332 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:36.353102 13435 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=41, attempt_no=0}) took 1879 ms. Trace:
I20240506 11:39:36.353228 13435 rpcz_store.cc:276] 0506 11:39:34.473593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:34.473663 (+ 70us) service_pool.cc:225] Handling call
0506 11:39:36.353044 (+1879381us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:34.933090 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:34.933215 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:34.933221 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:34.933223 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:34.933327 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:34.933337 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:34.933340 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:34.933390 (+ 50us) tablet.cc:699] Row locks acquired
0506 11:39:34.933392 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:34.933434 (+ 42us) write_op.cc:270] Start()
0506 11:39:35.730596 (+797162us) write_op.cc:276] Timestamp: P: 1714995574933380 usec, L: 0
0506 11:39:35.730601 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:35.730871 (+ 270us) log.cc:842] Serialized 354 byte log entry
0506 11:39:36.351996 (+621125us) op_driver.cc:444] REPLICATION: finished
0506 11:39:36.352459 (+ 463us) write_op.cc:301] APPLY: starting
0506 11:39:36.352476 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:36.352521 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:36.352549 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:36.352630 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:36.352654 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:36.352658 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:36.352759 (+ 101us) log.cc:842] Serialized 22 byte log entry
0506 11:39:36.352908 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:36.352927 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:36.352935 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":395,"num_ops":1,"prepare.queue_time_us":459037,"prepare.run_cpu_time_us":792,"prepare.run_wall_time_us":797849,"replication_time_us":621359,"thread_start_us":278,"threads_started":1}]]}
W20240506 11:39:36.355232 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.676s user 0.000s sys 0.001s
W20240506 11:39:36.355448 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.677s user 0.001s sys 0.000s
I20240506 11:39:36.513134 13341 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:36.513195 13017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:39005 (request call id 2941) took 1221 ms. Trace:
I20240506 11:39:36.513332 13341 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:36.514053 13341 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:36.513381 13017 rpcz_store.cc:276] 0506 11:39:35.291546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:35.291602 (+ 56us) service_pool.cc:225] Handling call
0506 11:39:35.291837 (+ 235us) raft_consensus.cc:1457] Updating replica for 3 ops
0506 11:39:35.291951 (+ 114us) raft_consensus.cc:1520] Early marking committed up to index 39
0506 11:39:35.292294 (+ 343us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0506 11:39:35.292415 (+ 121us) write_op.cc:270] Start()
0506 11:39:35.446562 (+154147us) write_op.cc:276] Timestamp: P: 1714995573197196 usec, L: 0
0506 11:39:35.446848 (+ 286us) write_op.cc:270] Start()
0506 11:39:36.283013 (+836165us) write_op.cc:276] Timestamp: P: 1714995573599936 usec, L: 0
0506 11:39:36.283579 (+ 566us) write_op.cc:270] Start()
0506 11:39:36.511734 (+228155us) write_op.cc:276] Timestamp: P: 1714995574357488 usec, L: 0
0506 11:39:36.512068 (+ 334us) log.cc:842] Serialized 1062 byte log entry
0506 11:39:36.512396 (+ 328us) raft_consensus.cc:1631] Marking committed up to 41
0506 11:39:36.513014 (+ 618us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:36.513042 (+ 28us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:36.513051 (+ 9us) raft_consensus.cc:1671] finished
0506 11:39:36.513055 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:36.513139 (+ 84us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:35.446771 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:35.446882 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:35.446886 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:35.446888 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:35.446992 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:35.447006 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:35.447009 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:35.447053 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:39:35.447055 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:36.512433 (+1065378us) op_driver.cc:444] REPLICATION: finished
0506 11:39:36.512770 (+ 337us) write_op.cc:301] APPLY: starting
0506 11:39:36.512783 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:36.512821 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:36.512822 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:36.512897 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:36.512941 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:36.512998 (+ 57us) write_op.cc:312] APPLY: finished
0506 11:39:36.513093 (+ 95us) log.cc:842] Serialized 22 byte log entry
0506 11:39:36.513166 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:36.513185 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:36.513191 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:39:36.513263 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:36.513268 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:39:36.283516 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:36.283624 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:36.283628 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:36.283629 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:36.283701 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:36.283711 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:36.283714 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:36.283758 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:39:36.283759 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:36.512715 (+228956us) op_driver.cc:444] REPLICATION: finished
0506 11:39:36.513135 (+ 420us) write_op.cc:301] APPLY: starting
0506 11:39:36.513147 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:36.513174 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:36.513176 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:36.513230 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:36.513253 (+ 23us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:36.513317 (+ 64us) write_op.cc:312] APPLY: finished
0506 11:39:36.513376 (+ 59us) log.cc:842] Serialized 22 byte log entry
0506 11:39:36.513413 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:36.513422 (+ 9us) write_op.cc:454] Released schema lock
0506 11:39:36.513426 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0506 11:39:36.513504 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:36.513507 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:39:36.511933 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:36.512018 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:36.512023 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:36.512025 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:36.512115 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:36.512126 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:36.512129 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:36.512172 (+ 43us) tablet.cc:699] Row locks acquired
0506 11:39:36.512173 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":247,"threads_started":1,"wal-append.queue_time_us":336,"child_traces":[["op",{"apply.queue_time_us":284,"apply.run_cpu_time_us":557,"apply.run_wall_time_us":556,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":72,"prepare.run_cpu_time_us":375,"prepare.run_wall_time_us":374,"replication_time_us":1220003,"thread_start_us":205,"threads_started":1}],["op",{"apply.queue_time_us":341,"apply.run_cpu_time_us":410,"apply.run_wall_time_us":411,"mutex_wait_us":22,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":377,"prepare.run_cpu_time_us":319,"prepare.run_wall_time_us":320,"replication_time_us":1065855,"thread_start_us":474,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":64,"prepare.run_cpu_time_us":303,"prepare.run_wall_time_us":303}]]}
I20240506 11:39:36.514518 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:36.514539 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:36.515144 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:36.515486 13341 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:37.043123 13420 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:37.078850 13443 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=42, attempt_no=0}) took 1551 ms. Trace:
I20240506 11:39:37.078991 13443 rpcz_store.cc:276] 0506 11:39:35.527460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:35.527508 (+ 48us) service_pool.cc:225] Handling call
0506 11:39:37.078793 (+1551285us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:35.730990 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:35.731103 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:35.731109 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:35.731111 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:35.731208 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:35.731219 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:35.731223 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:35.731297 (+ 74us) tablet.cc:699] Row locks acquired
0506 11:39:35.731299 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:35.731343 (+ 44us) write_op.cc:270] Start()
0506 11:39:35.908529 (+177186us) write_op.cc:276] Timestamp: P: 1714995575731315 usec, L: 0
0506 11:39:35.908534 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:35.908780 (+ 246us) log.cc:842] Serialized 354 byte log entry
0506 11:39:37.077797 (+1169017us) op_driver.cc:444] REPLICATION: finished
0506 11:39:37.078251 (+ 454us) write_op.cc:301] APPLY: starting
0506 11:39:37.078268 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:37.078333 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:37.078368 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:37.078462 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:37.078488 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:37.078493 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:37.078578 (+ 85us) log.cc:842] Serialized 22 byte log entry
0506 11:39:37.078663 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:37.078682 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:37.078689 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":383,"num_ops":1,"prepare.queue_time_us":202998,"prepare.run_cpu_time_us":800,"prepare.run_wall_time_us":177874,"replication_time_us":1169211,"thread_start_us":277,"threads_started":1}]]}
I20240506 11:39:37.265470 13420 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:37.265671 13420 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:37.266484 13420 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:37.266865 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:37.266893 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:37.267537 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:37.267843 13420 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 11:39:37.287433 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.781s user 0.001s sys 0.000s
W20240506 11:39:37.287474 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.782s user 0.000s sys 0.001s
I20240506 11:39:37.572577 13017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:39005 (request call id 2942) took 1057 ms. Trace:
I20240506 11:39:37.572728 13017 rpcz_store.cc:276] 0506 11:39:36.515490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:36.515565 (+ 75us) service_pool.cc:225] Handling call
0506 11:39:36.515779 (+ 214us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 11:39:36.515898 (+ 119us) raft_consensus.cc:1520] Early marking committed up to index 42
0506 11:39:36.515958 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 11:39:36.516041 (+ 83us) write_op.cc:270] Start()
0506 11:39:36.726205 (+210164us) write_op.cc:276] Timestamp: P: 1714995574933380 usec, L: 0
0506 11:39:36.726593 (+ 388us) write_op.cc:270] Start()
0506 11:39:37.570750 (+844157us) write_op.cc:276] Timestamp: P: 1714995575731315 usec, L: 0
0506 11:39:37.571359 (+ 609us) log.cc:842] Serialized 708 byte log entry
0506 11:39:37.571670 (+ 311us) raft_consensus.cc:1631] Marking committed up to 43
0506 11:39:37.572414 (+ 744us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:37.572441 (+ 27us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:37.572455 (+ 14us) raft_consensus.cc:1671] finished
0506 11:39:37.572457 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:37.572504 (+ 47us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:36.726450 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:36.726591 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:36.726597 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:36.726599 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:36.726698 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:36.726732 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:36.726738 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:36.726793 (+ 55us) tablet.cc:699] Row locks acquired
0506 11:39:36.726794 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:37.571724 (+844930us) op_driver.cc:444] REPLICATION: finished
0506 11:39:37.572473 (+ 749us) write_op.cc:301] APPLY: starting
0506 11:39:37.572489 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:37.572531 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:37.572533 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:37.572633 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:37.572667 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:37.572702 (+ 35us) write_op.cc:312] APPLY: finished
0506 11:39:37.572770 (+ 68us) log.cc:842] Serialized 22 byte log entry
Related trace 'op':
0506 11:39:37.571217 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:37.571318 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:37.571323 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:37.571325 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:37.571433 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:37.571451 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:37.571456 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:37.571546 (+ 90us) tablet.cc:699] Row locks acquired
0506 11:39:37.571551 (+ 5us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":234,"threads_started":1,"wal-append.queue_time_us":314,"child_traces":[["op",{"apply.queue_time_us":692,"apply.run_cpu_time_us":509,"apply.run_wall_time_us":669,"num_ops":1,"partition_lock_wait_us":24,"prepare.queue_time_us":71,"prepare.run_cpu_time_us":451,"prepare.run_wall_time_us":453,"replication_time_us":1055653,"thread_start_us":615,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":335,"prepare.run_cpu_time_us":443,"prepare.run_wall_time_us":444,"thread_start_us":218,"threads_started":1}]]}
I20240506 11:39:37.573310 13341 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:37.573499 13341 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:37.574227 13341 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:37.574482 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:37.574913 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:37.575553 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:38.039747 13341 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:38.066514 13341 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:38.066723 13341 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:38.067540 13341 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:38.068099 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:38.068133 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:38.068828 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:38.069059 13341 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:38.109362 13448 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=44, attempt_no=0}) took 1028 ms. Trace:
I20240506 11:39:38.109557 13448 rpcz_store.cc:276] 0506 11:39:37.080644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:37.080699 (+ 55us) service_pool.cc:225] Handling call
0506 11:39:38.109242 (+1028543us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:37.081130 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:37.081223 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:37.081227 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:37.081228 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:37.081290 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:37.081298 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:37.081300 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:37.081335 (+ 35us) tablet.cc:699] Row locks acquired
0506 11:39:37.081336 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:37.081365 (+ 29us) write_op.cc:270] Start()
0506 11:39:37.989518 (+908153us) write_op.cc:276] Timestamp: P: 1714995577081335 usec, L: 0
0506 11:39:37.989524 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:39:37.989842 (+ 318us) log.cc:842] Serialized 354 byte log entry
0506 11:39:38.106657 (+116815us) op_driver.cc:444] REPLICATION: finished
0506 11:39:38.107464 (+ 807us) write_op.cc:301] APPLY: starting
0506 11:39:38.107479 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:38.107514 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:38.107545 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:38.107646 (+ 101us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:38.107749 (+ 103us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:38.107757 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:39:38.107909 (+ 152us) log.cc:842] Serialized 22 byte log entry
0506 11:39:38.109111 (+ 1202us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:38.109133 (+ 22us) write_op.cc:454] Released schema lock
0506 11:39:38.109141 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":740,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":794,"prepare.run_wall_time_us":908865,"raft.queue_time_us":67,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":117095,"spinlock_wait_cycles":62848,"thread_start_us":644,"threads_started":1}]]}
W20240506 11:39:38.110590 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.672s user 0.000s sys 0.000s
W20240506 11:39:38.110553 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.672s user 0.001s sys 0.000s
I20240506 11:39:38.840090 13447 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:38.840327 13447 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:38.841202 13447 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:38.841708 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:38.841907 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:38.842450 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:39.301033 13341 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:39.301218 13341 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:39.302009 13341 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:39.302407 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:39.303354 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:39.303879 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:39.351419 13456 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=45, attempt_no=0}) took 2063 ms. Trace:
I20240506 11:39:39.351570 13456 rpcz_store.cc:276] 0506 11:39:37.288028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:37.288083 (+ 55us) service_pool.cc:225] Handling call
0506 11:39:39.351345 (+2063262us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:37.990035 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:37.990179 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:37.990186 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:39:37.990189 (+ 3us) tablet.cc:660] Decoding operations
0506 11:39:37.990286 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:37.990299 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:37.990304 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:37.990367 (+ 63us) tablet.cc:699] Row locks acquired
0506 11:39:37.990370 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:39:37.990412 (+ 42us) write_op.cc:270] Start()
0506 11:39:38.434561 (+444149us) write_op.cc:276] Timestamp: P: 1714995577990372 usec, L: 0
0506 11:39:38.434566 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:38.434835 (+ 269us) log.cc:842] Serialized 354 byte log entry
0506 11:39:39.350402 (+915567us) op_driver.cc:444] REPLICATION: finished
0506 11:39:39.350804 (+ 402us) write_op.cc:301] APPLY: starting
0506 11:39:39.350822 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:39.350856 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:39.350885 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:39.350973 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:39.351003 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:39.351008 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:39.351094 (+ 86us) log.cc:842] Serialized 22 byte log entry
0506 11:39:39.351202 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:39.351222 (+ 20us) write_op.cc:454] Released schema lock
0506 11:39:39.351231 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":331,"num_ops":1,"prepare.queue_time_us":701605,"prepare.run_cpu_time_us":811,"prepare.run_wall_time_us":444884,"raft.queue_time_us":36,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":220,"replication_time_us":915799,"thread_start_us":217,"threads_started":1}]]}
I20240506 11:39:39.353024 12845 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 105) took 1091 ms. Trace:
I20240506 11:39:39.353078 12846 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 106) took 1090 ms. Trace:
I20240506 11:39:39.353129 12845 rpcz_store.cc:276] 0506 11:39:38.261977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:38.262040 (+ 63us) service_pool.cc:225] Handling call
0506 11:39:38.262265 (+ 225us) tablet_service.cc:2780] Created scanner 9c643b9741b74c8f87b43ea7beece06a for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 16448d7ad341499ea3be607c7be89abc
0506 11:39:38.263136 (+ 871us) tablet_service.cc:2920] Creating iterator
0506 11:39:38.263163 (+ 27us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:38.434706 (+171543us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:39.351245 (+916539us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1088066 microseconds
0506 11:39:39.351390 (+ 145us) tablet_service.cc:2945] Iterator created
0506 11:39:39.351928 (+ 538us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:39.351956 (+ 28us) tablet_service.cc:3010] has_more: true
0506 11:39:39.351992 (+ 36us) tablet_service.cc:3027] Continuing scan request
0506 11:39:39.352063 (+ 71us) tablet_service.cc:3091] Found scanner 9c643b9741b74c8f87b43ea7beece06a for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 16448d7ad341499ea3be607c7be89abc
0506 11:39:39.352970 (+ 907us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20240506 11:39:39.353189 12846 rpcz_store.cc:276] 0506 11:39:38.262174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:38.262238 (+ 64us) service_pool.cc:225] Handling call
0506 11:39:38.262400 (+ 162us) tablet_service.cc:2780] Created scanner d6527758ddef4848940783dc831e9297 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 4cdb09cd73a546c99997afce0673ae10
0506 11:39:38.263200 (+ 800us) tablet_service.cc:2920] Creating iterator
0506 11:39:38.263225 (+ 25us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:38.434717 (+171492us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:39.351290 (+916573us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1088056 microseconds
0506 11:39:39.351394 (+ 104us) tablet_service.cc:2945] Iterator created
0506 11:39:39.352041 (+ 647us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:39.352070 (+ 29us) tablet_service.cc:3010] has_more: true
0506 11:39:39.352111 (+ 41us) tablet_service.cc:3027] Continuing scan request
0506 11:39:39.352188 (+ 77us) tablet_service.cc:3091] Found scanner d6527758ddef4848940783dc831e9297 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 4cdb09cd73a546c99997afce0673ae10
0506 11:39:39.353023 (+ 835us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":6016}
W20240506 11:39:39.353837 13239 scanner-internal.cc:458] Time spent opening tablet: real 1.093s user 0.000s sys 0.000s
W20240506 11:39:39.353881 13240 scanner-internal.cc:458] Time spent opening tablet: real 1.093s user 0.001s sys 0.000s
I20240506 11:39:39.477231 13341 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:39.519129 13456 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=46, attempt_no=0}) took 1407 ms. Trace:
I20240506 11:39:39.519297 13456 rpcz_store.cc:276] 0506 11:39:38.112059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:38.112101 (+ 42us) service_pool.cc:225] Handling call
0506 11:39:39.519062 (+1406961us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:38.434969 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:38.435084 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:38.435092 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:39:38.435095 (+ 3us) tablet.cc:660] Decoding operations
0506 11:39:38.435196 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:38.435208 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:38.435212 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:38.435280 (+ 68us) tablet.cc:699] Row locks acquired
0506 11:39:38.435283 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:39:38.435329 (+ 46us) write_op.cc:270] Start()
0506 11:39:38.819471 (+384142us) write_op.cc:276] Timestamp: P: 1714995578435294 usec, L: 0
0506 11:39:38.819475 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:38.819696 (+ 221us) log.cc:842] Serialized 354 byte log entry
0506 11:39:39.518232 (+698536us) op_driver.cc:444] REPLICATION: finished
0506 11:39:39.518340 (+ 108us) write_op.cc:301] APPLY: starting
0506 11:39:39.518359 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:39.518402 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:39.518436 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:39.518520 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:39.518554 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:39.518562 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:39:39.518671 (+ 109us) log.cc:842] Serialized 22 byte log entry
0506 11:39:39.518923 (+ 252us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:39.518942 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:39.518950 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46,"num_ops":1,"prepare.queue_time_us":322473,"prepare.run_cpu_time_us":757,"prepare.run_wall_time_us":384820,"replication_time_us":698730}]]}
I20240506 11:39:39.914376 13447 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:39.915253 13146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:57675 (request call id 5622) took 1073 ms. Trace:
I20240506 11:39:39.915381 13146 rpcz_store.cc:276] 0506 11:39:38.841494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:38.841635 (+ 141us) service_pool.cc:225] Handling call
0506 11:39:38.841851 (+ 216us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 11:39:38.841981 (+ 130us) raft_consensus.cc:1520] Early marking committed up to index 46
0506 11:39:38.842475 (+ 494us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 11:39:38.842600 (+ 125us) write_op.cc:270] Start()
0506 11:39:38.999783 (+157183us) write_op.cc:276] Timestamp: P: 1714995577990372 usec, L: 0
0506 11:39:39.000129 (+ 346us) write_op.cc:270] Start()
0506 11:39:39.913285 (+913156us) write_op.cc:276] Timestamp: P: 1714995578435294 usec, L: 0
0506 11:39:39.913911 (+ 626us) log.cc:842] Serialized 708 byte log entry
0506 11:39:39.914280 (+ 369us) raft_consensus.cc:1631] Marking committed up to 46
0506 11:39:39.914287 (+ 7us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:39.914322 (+ 35us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:39.915151 (+ 829us) raft_consensus.cc:1671] finished
0506 11:39:39.915156 (+ 5us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:39.915201 (+ 45us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:39.000026 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:39.000146 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:39.000152 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:39.000153 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:39.000226 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:39.000248 (+ 22us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:39.000251 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:39.000292 (+ 41us) tablet.cc:699] Row locks acquired
0506 11:39:39.000293 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0506 11:39:39.913807 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:39.913912 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:39.913917 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:39.913918 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:39.913996 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:39.914007 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:39.914010 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:39.914054 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:39:39.914056 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":281,"threads_started":1,"wal-append.queue_time_us":363,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":81,"prepare.run_cpu_time_us":362,"prepare.run_wall_time_us":362}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":398,"prepare.run_cpu_time_us":332,"prepare.run_wall_time_us":333,"thread_start_us":255,"threads_started":1}]]}
I20240506 11:39:40.077487 13463 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:40.077740 13463 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:40.079001 13463 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:40.079062 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:40.079138 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:40.079717 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:40.079978 13463 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:40.350961 13447 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:40.351166 13447 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:40.351949 13447 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:40.352396 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:40.352403 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:40.352993 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:40.353243 13447 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:40.371395 13464 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=47, attempt_no=0}) took 1017 ms. Trace:
I20240506 11:39:40.371552 13464 rpcz_store.cc:276] 0506 11:39:39.353883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:39.353943 (+ 60us) service_pool.cc:225] Handling call
0506 11:39:40.371336 (+1017393us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:39.354652 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:39.354763 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:39.354767 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:39.354769 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:39.354859 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:39.354868 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:39.354871 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:39.354915 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:39:39.354917 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:39.354979 (+ 62us) write_op.cc:270] Start()
0506 11:39:40.194132 (+839153us) write_op.cc:276] Timestamp: P: 1714995579354931 usec, L: 0
0506 11:39:40.194137 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:40.194384 (+ 247us) log.cc:842] Serialized 354 byte log entry
0506 11:39:40.370256 (+175872us) op_driver.cc:444] REPLICATION: finished
0506 11:39:40.370639 (+ 383us) write_op.cc:301] APPLY: starting
0506 11:39:40.370656 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:40.370710 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:40.370743 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:40.370831 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:40.370857 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:40.370861 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:40.371000 (+ 139us) log.cc:842] Serialized 22 byte log entry
0506 11:39:40.371170 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:40.371187 (+ 17us) write_op.cc:454] Released schema lock
0506 11:39:40.371194 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":319,"num_ops":1,"prepare.queue_time_us":309,"prepare.run_cpu_time_us":733,"prepare.run_wall_time_us":839803,"replication_time_us":176086,"thread_start_us":446,"threads_started":2}]]}
W20240506 11:39:40.373422 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.869s user 0.000s sys 0.000s
W20240506 11:39:40.374159 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.870s user 0.001s sys 0.000s
I20240506 11:39:40.553478 13463 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:40.553699 13463 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:40.554421 13463 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:40.554884 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:40.555012 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:40.555833 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:40.556238 13463 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:40.689308 13447 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:40.689529 13447 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:40.690198 13447 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:40.690936 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:40.691435 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:40.691977 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:40.692211 13447 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:41.089010 13463 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:41.089205 13463 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:41.089965 13463 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:41.090469 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:41.090514 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:41.091048 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:41.091387 13463 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:41.098167 13470 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=48, attempt_no=0}) took 1577 ms. Trace:
I20240506 11:39:41.098320 13470 rpcz_store.cc:276] 0506 11:39:39.520981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:39.521042 (+ 61us) service_pool.cc:225] Handling call
0506 11:39:41.098107 (+1577065us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:40.194489 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:40.194631 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:40.194637 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:40.194639 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:40.194762 (+ 123us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:40.194773 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:40.194778 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:40.194831 (+ 53us) tablet.cc:699] Row locks acquired
0506 11:39:40.194832 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:40.194867 (+ 35us) write_op.cc:270] Start()
0506 11:39:40.960974 (+766107us) write_op.cc:276] Timestamp: P: 1714995580194832 usec, L: 0
0506 11:39:40.960981 (+ 7us) op_driver.cc:331] REPLICATION: starting
0506 11:39:40.961278 (+ 297us) log.cc:842] Serialized 354 byte log entry
0506 11:39:41.096999 (+135721us) op_driver.cc:444] REPLICATION: finished
0506 11:39:41.097500 (+ 501us) write_op.cc:301] APPLY: starting
0506 11:39:41.097515 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:41.097551 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:41.097570 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:41.097693 (+ 123us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:41.097724 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:41.097731 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:39:41.097857 (+ 126us) log.cc:842] Serialized 22 byte log entry
0506 11:39:41.097980 (+ 123us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:41.097996 (+ 16us) write_op.cc:454] Released schema lock
0506 11:39:41.098009 (+ 13us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":409,"num_ops":1,"prepare.queue_time_us":673095,"prepare.run_cpu_time_us":901,"prepare.run_wall_time_us":766927,"raft.queue_time_us":213,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":135990,"spinlock_wait_cycles":64896,"thread_start_us":296,"threads_started":1}]]}
W20240506 11:39:41.100917 13240 scanner-internal.cc:458] Time spent opening tablet: real 0.576s user 0.001s sys 0.000s
W20240506 11:39:41.101557 13239 scanner-internal.cc:458] Time spent opening tablet: real 0.577s user 0.000s sys 0.000s
I20240506 11:39:41.265246 13447 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:41.265429 13447 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:41.266204 13447 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:41.266827 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:41.266877 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:41.267486 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:41.267957 13447 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:42.287003 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:42.287264 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:42.288300 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:42.288753 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:42.288764 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:42.289403 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:42.289664 13475 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:42.320297 13479 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=49, attempt_no=0}) took 1947 ms. Trace:
I20240506 11:39:42.320506 13479 rpcz_store.cc:276] 0506 11:39:40.373287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:40.373338 (+ 51us) service_pool.cc:225] Handling call
0506 11:39:42.320272 (+1946934us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:40.961470 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:40.961595 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:40.961602 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:39:40.961605 (+ 3us) tablet.cc:660] Decoding operations
0506 11:39:40.961716 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:40.961729 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:40.961733 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:40.961791 (+ 58us) tablet.cc:699] Row locks acquired
0506 11:39:40.961793 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:40.961841 (+ 48us) write_op.cc:270] Start()
0506 11:39:41.918992 (+957151us) write_op.cc:276] Timestamp: P: 1714995580961811 usec, L: 0
0506 11:39:41.918996 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:41.919275 (+ 279us) log.cc:842] Serialized 354 byte log entry
0506 11:39:42.319241 (+399966us) op_driver.cc:444] REPLICATION: finished
0506 11:39:42.319704 (+ 463us) write_op.cc:301] APPLY: starting
0506 11:39:42.319724 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:42.319767 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:42.319793 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:42.319870 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:42.319906 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:42.319914 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:39:42.320029 (+ 115us) log.cc:842] Serialized 22 byte log entry
0506 11:39:42.320147 (+ 118us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:42.320166 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:42.320173 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":403,"num_ops":1,"prepare.queue_time_us":587672,"prepare.run_cpu_time_us":829,"prepare.run_wall_time_us":957901,"raft.queue_time_us":259,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":251,"replication_time_us":400217,"spinlock_wait_cycles":80640,"thread_start_us":285,"threads_started":1}]]}
I20240506 11:39:42.321915 12846 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 115) took 1069 ms. Trace:
I20240506 11:39:42.322016 12846 rpcz_store.cc:276] 0506 11:39:41.252223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:41.252296 (+ 73us) service_pool.cc:225] Handling call
0506 11:39:41.252511 (+ 215us) tablet_service.cc:2780] Created scanner 68ee6bcb9e09470787a19debd492cf63 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 7a40ca0138484bc0a2ea1d5a8e0ed7bc
0506 11:39:41.253279 (+ 768us) tablet_service.cc:2920] Creating iterator
0506 11:39:41.253306 (+ 27us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:41.919111 (+665805us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:42.320182 (+401071us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1066861 microseconds
0506 11:39:42.320293 (+ 111us) tablet_service.cc:2945] Iterator created
0506 11:39:42.320935 (+ 642us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:42.320976 (+ 41us) tablet_service.cc:3010] has_more: true
0506 11:39:42.321018 (+ 42us) tablet_service.cc:3027] Continuing scan request
0506 11:39:42.321094 (+ 76us) tablet_service.cc:3091] Found scanner 68ee6bcb9e09470787a19debd492cf63 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is 7a40ca0138484bc0a2ea1d5a8e0ed7bc
0506 11:39:42.321897 (+ 803us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":23552}
I20240506 11:39:42.322582 12845 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:39170 (request call id 116) took 1070 ms. Trace:
I20240506 11:39:42.322659 12845 rpcz_store.cc:276] 0506 11:39:41.252546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:41.252593 (+ 47us) service_pool.cc:225] Handling call
0506 11:39:41.252758 (+ 165us) tablet_service.cc:2780] Created scanner 7ea022945fe6442089882a26fa1014e3 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is f58c99db9ab34cb2b0a089cced63a15f
0506 11:39:41.253435 (+ 677us) tablet_service.cc:2920] Creating iterator
0506 11:39:41.253458 (+ 23us) tablet_service.cc:3297] Waiting safe time to advance
0506 11:39:41.919136 (+665678us) tablet_service.cc:3304] Waiting for operations to commit
0506 11:39:42.320182 (+401046us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1066709 microseconds
0506 11:39:42.320293 (+ 111us) tablet_service.cc:2945] Iterator created
0506 11:39:42.320992 (+ 699us) tablet_service.cc:2967] Iterator init: OK
0506 11:39:42.321011 (+ 19us) tablet_service.cc:3010] has_more: true
0506 11:39:42.321032 (+ 21us) tablet_service.cc:3027] Continuing scan request
0506 11:39:42.321094 (+ 62us) tablet_service.cc:3091] Found scanner 7ea022945fe6442089882a26fa1014e3 for tablet 831a913fdab244808c7b4bae4a4a56c3, query id is f58c99db9ab34cb2b0a089cced63a15f
0506 11:39:42.322567 (+ 1473us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":2048}
W20240506 11:39:42.322846 13240 scanner-internal.cc:458] Time spent opening tablet: real 1.071s user 0.001s sys 0.000s
W20240506 11:39:42.323156 13239 scanner-internal.cc:458] Time spent opening tablet: real 1.071s user 0.000s sys 0.000s
I20240506 11:39:42.436962 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:42.437237 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:42.438206 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:42.438627 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:42.439208 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:42.439811 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:42.440096 13474 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:42.649407 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:42.649595 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:42.650215 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:42.650789 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:42.651278 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:42.651927 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:42.652195 13475 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:42.688445 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:42.688617 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:42.689211 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:42.689956 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:42.690176 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:42.690929 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:42.691421 13474 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:43.147449 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:43.147665 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.148458 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.148965 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:43.148983 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:43.149561 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:43.150180 13475 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:43.193480 13485 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39170 (ReqId={client: 020f44488ef74eb6841d078304e929a7, seq_no=50, attempt_no=0}) took 2091 ms. Trace:
I20240506 11:39:43.193603 13485 rpcz_store.cc:276] 0506 11:39:41.101566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:41.101631 (+ 65us) service_pool.cc:225] Handling call
0506 11:39:43.193457 (+2091826us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:41.919413 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 831a913fdab244808c7b4bae4a4a56c3
0506 11:39:41.919533 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:41.919539 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:41.919541 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:41.919658 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:41.919670 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:41.919675 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:41.919731 (+ 56us) tablet.cc:699] Row locks acquired
0506 11:39:41.919733 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:41.919776 (+ 43us) write_op.cc:270] Start()
0506 11:39:42.894905 (+975129us) write_op.cc:276] Timestamp: P: 1714995581919733 usec, L: 0
0506 11:39:42.894909 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:42.895133 (+ 224us) log.cc:842] Serialized 354 byte log entry
0506 11:39:43.192477 (+297344us) op_driver.cc:444] REPLICATION: finished
0506 11:39:43.192897 (+ 420us) write_op.cc:301] APPLY: starting
0506 11:39:43.192911 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:43.192944 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:43.192964 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:43.193022 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:43.193057 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:43.193061 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:39:43.193187 (+ 126us) log.cc:842] Serialized 22 byte log entry
0506 11:39:43.193272 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:43.193332 (+ 60us) write_op.cc:454] Released schema lock
0506 11:39:43.193341 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":348,"num_ops":1,"prepare.queue_time_us":817345,"prepare.run_cpu_time_us":824,"prepare.run_wall_time_us":975877,"raft.queue_time_us":25,"raft.run_cpu_time_us":36,"raft.run_wall_time_us":37,"replication_time_us":297542,"spinlock_wait_cycles":141568,"thread_start_us":231,"threads_started":1}]]}
I20240506 11:39:43.280071 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:43.280284 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.281147 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.281497 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:43.281529 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:43.282882 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:43.283154 13475 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:43.296566 12866 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240506 11:39:43.297472 12997 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:39:43.748502 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:43.748732 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.749485 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.750125 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:43.750543 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:43.751070 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:43.752228 13127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:39:43.752480 13474 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:43.760221 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:43.760365 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.760938 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.761274 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:43.761425 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:43.762010 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:43.762290 13475 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7851e30a07e642cc844a5c32667d8ce7 | 127.10.45.190:43667 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:36329 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
30ffd8145bbd483bbb01f029813c4541 | 127.10.45.130:41551 | HEALTHY | <none> | 0 | 0
4d74671f77b041ae90c1cb2bc6d6c5ab | 127.10.45.129:33081 | HEALTHY | <none> | 1 | 0
89b865272ad043919295f14ca4025cf6 | 127.10.45.131:32893 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
consensus_inject_latency_ms_in_notifications | 50 | unsafe,hidden | all 3 server(s) checked
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.45.129 | experimental | 127.10.45.129:33081
local_ip_for_outbound_sockets | 127.10.45.130 | experimental | 127.10.45.130:41551
local_ip_for_outbound_sockets | 127.10.45.131 | experimental | 127.10.45.131:32893
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.45.129:33081
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.45.130:41551
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.45.131:32893
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:36329 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240506 11:39:43.797073 10422 log_verifier.cc:126] Checking tablet 831a913fdab244808c7b4bae4a4a56c3
I20240506 11:39:43.855439 10422 log_verifier.cc:177] Verified matching terms for 52 ops in tablet 831a913fdab244808c7b4bae4a4a56c3
I20240506 11:39:43.882233 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 12804
I20240506 11:39:43.893040 12926 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:39:43.900456 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:43.900617 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.901216 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.901547 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:43.901582 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:43.901769 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.902089 13146 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ffd8145bbd483bbb01f029813c4541 in term 1.
I20240506 11:39:43.902360 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.902683 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:43.902679 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:43.902927 13017 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89b865272ad043919295f14ca4025cf6 in term 1.
I20240506 11:39:43.903065 13475 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 11:39:43.903167 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:43.903245 13475 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:39:43.903370 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:43.903591 13474 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 11:39:43.903694 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d74671f77b041ae90c1cb2bc6d6c5ab)
I20240506 11:39:43.903777 13474 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:39:43.905462 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.905845 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.906033 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 election: Requested vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.906487 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 election: Requested vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.906646 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541"
I20240506 11:39:43.906848 13017 raft_consensus.cc:2379] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 89b865272ad043919295f14ca4025cf6 in current term 2: Already voted for candidate 30ffd8145bbd483bbb01f029813c4541 in this term.
I20240506 11:39:43.906877 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6"
I20240506 11:39:43.907142 13146 raft_consensus.cc:2379] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ffd8145bbd483bbb01f029813c4541 in current term 2: Already voted for candidate 89b865272ad043919295f14ca4025cf6 in this term.
W20240506 11:39:43.907742 12953 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.006s (SENT) [suppressed 1 similar messages]
W20240506 11:39:43.908581 13083 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.006s (SENT) [suppressed 1 similar messages]
W20240506 11:39:43.914819 12953 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.006s (SENT)
W20240506 11:39:43.916195 13083 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.006s (SENT)
W20240506 11:39:43.923703 13083 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.008s (SENT)
I20240506 11:39:43.923877 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
W20240506 11:39:43.924156 12953 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.008s (SENT)
I20240506 11:39:43.924237 13474 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 11:39:43.924273 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:43.924582 13475 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 11:39:43.931277 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:43.931442 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.931931 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:43.932116 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.932102 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.932597 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 3 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:43.932715 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.932835 13146 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ffd8145bbd483bbb01f029813c4541 in term 2.
I20240506 11:39:43.933210 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:43.933233 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:43.933446 13017 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89b865272ad043919295f14ca4025cf6 in term 2.
I20240506 11:39:43.933590 13475 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 11:39:43.933730 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:43.933789 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:43.933840 13475 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 2 FOLLOWER]: Advancing to term 3
I20240506 11:39:43.934053 13474 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 11:39:43.934159 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:43.934260 13474 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 2 FOLLOWER]: Advancing to term 3
I20240506 11:39:43.936625 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.936772 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.937484 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 3 election: Requested vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.937628 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 3 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6"
I20240506 11:39:43.937919 13146 raft_consensus.cc:2379] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ffd8145bbd483bbb01f029813c4541 in current term 3: Already voted for candidate 89b865272ad043919295f14ca4025cf6 in this term.
I20240506 11:39:43.938112 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541"
I20240506 11:39:43.938416 13017 raft_consensus.cc:2379] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 89b865272ad043919295f14ca4025cf6 in current term 3: Already voted for candidate 30ffd8145bbd483bbb01f029813c4541 in this term.
I20240506 11:39:43.937441 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 3 election: Requested vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
W20240506 11:39:43.945168 12953 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.006s (SENT)
W20240506 11:39:43.947633 13083 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.007s (SENT)
W20240506 11:39:43.958285 12953 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.010s (SENT)
I20240506 11:39:43.958431 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:43.958757 13475 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240506 11:39:43.960700 13083 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.011s (SENT)
I20240506 11:39:43.960855 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:43.961160 13474 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 11:39:43.971042 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:43.971217 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.971750 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:43.971863 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.971995 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.972375 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:43.972707 13017 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89b865272ad043919295f14ca4025cf6 in term 3.
I20240506 11:39:43.973084 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.973253 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:43.973356 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 4 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:43.973515 13474 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 11:39:43.973676 13146 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ffd8145bbd483bbb01f029813c4541 in term 3.
I20240506 11:39:43.973845 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:43.974022 12950 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:43.974143 13474 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 3 FOLLOWER]: Advancing to term 4
I20240506 11:39:43.974268 13475 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 11:39:43.974359 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:43.974440 13475 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 3 FOLLOWER]: Advancing to term 4
I20240506 11:39:43.976189 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.976727 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 4 election: Requested vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.976962 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:43.977070 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 4 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6"
I20240506 11:39:43.977401 13146 raft_consensus.cc:2379] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 30ffd8145bbd483bbb01f029813c4541 in current term 4: Already voted for candidate 89b865272ad043919295f14ca4025cf6 in this term.
I20240506 11:39:43.977692 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 4 election: Requested vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:43.978057 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541"
I20240506 11:39:43.978271 13017 raft_consensus.cc:2379] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 89b865272ad043919295f14ca4025cf6 in current term 4: Already voted for candidate 30ffd8145bbd483bbb01f029813c4541 in this term.
W20240506 11:39:43.998253 12953 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.012s (SENT)
W20240506 11:39:44.004848 13083 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.013s (SENT)
I20240506 11:39:44.004971 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89b865272ad043919295f14ca4025cf6; no voters: 30ffd8145bbd483bbb01f029813c4541, 4d74671f77b041ae90c1cb2bc6d6c5ab
I20240506 11:39:44.005295 13474 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240506 11:39:44.010778 12953 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.016s (SENT)
I20240506 11:39:44.010910 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:44.011250 13475 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240506 11:39:44.011520 13083 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.019s (SENT)
I20240506 11:39:44.024780 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:44.024979 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:44.025602 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:44.026119 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:44.026319 13017 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89b865272ad043919295f14ca4025cf6 in term 4.
I20240506 11:39:44.026662 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:44.026893 13474 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 11:39:44.026988 13474 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:44.027104 13474 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 4 FOLLOWER]: Advancing to term 5
I20240506 11:39:44.028879 13474 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:44.029394 13474 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 5 election: Requested vote from peers 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:44.029814 13017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541"
I20240506 11:39:44.029989 13017 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 4 FOLLOWER]: Advancing to term 5
I20240506 11:39:44.031867 13017 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89b865272ad043919295f14ca4025cf6 in term 5.
I20240506 11:39:44.032230 13083 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541, 89b865272ad043919295f14ca4025cf6; no voters:
I20240506 11:39:44.032442 13474 raft_consensus.cc:2781] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 11:39:44.032912 13474 raft_consensus.cc:686] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 5 LEADER]: Becoming Leader. State: Replica: 89b865272ad043919295f14ca4025cf6, State: Running, Role: LEADER
I20240506 11:39:44.033424 13474 consensus_queue.cc:237] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.52, Last appended by leader: 52, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:44.036705 13017 raft_consensus.cc:1261] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 5 FOLLOWER]: Refusing update from remote peer 89b865272ad043919295f14ca4025cf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 5 index: 53. (index mismatch)
I20240506 11:39:44.036721 12745 catalog_manager.cc:5484] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 reported cstate change: term changed from 1 to 5, leader changed from 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129) to 89b865272ad043919295f14ca4025cf6 (127.10.45.131). New cstate: current_term: 5 leader_uuid: "89b865272ad043919295f14ca4025cf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } health_report { overall_health: UNKNOWN } } }
I20240506 11:39:44.037279 13529 consensus_queue.cc:1035] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
W20240506 11:39:44.047565 13083 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.008s (SENT)
W20240506 11:39:44.055393 13083 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.014s (SENT)
I20240506 11:39:44.091508 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
I20240506 11:39:44.092103 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
I20240506 11:39:44.092489 13146 raft_consensus.cc:1226] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 [term 5 LEADER]: Rejecting Update request from peer 4d74671f77b041ae90c1cb2bc6d6c5ab for earlier term 1. Current term is 5. Ops: []
I20240506 11:39:44.092490 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:44.093076 13523 consensus_queue.cc:1046] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 }, Status: INVALID_TERM, Last received: 1.52, Next index: 53, Last known committed idx: 53, Time since last communication: 0.000s
I20240506 11:39:44.093242 12882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
I20240506 11:39:44.093350 13523 consensus_queue.cc:1225] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [LEADER]: Peer 89b865272ad043919295f14ca4025cf6 log is divergent from this leader: its last log entry 5.53 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 53
I20240506 11:39:44.093608 12870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:44.094343 12871 raft_consensus.cc:3032] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:39:44.094412 12871 raft_consensus.cc:729] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d74671f77b041ae90c1cb2bc6d6c5ab, State: Running, Role: LEADER
I20240506 11:39:44.094694 12871 consensus_queue.cc:260] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.52, Last appended by leader: 52, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:44.095149 12871 raft_consensus.cc:3037] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 1 FOLLOWER]: Advancing to term 5
I20240506 11:39:44.095285 12886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:44.095683 12885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
I20240506 11:39:44.096145 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
W20240506 11:39:44.096396 12883 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.131:57337 (request call id 42) took 5 ms (client timeout 6 ms). Trace:
W20240506 11:39:44.096518 12883 rpcz_store.cc:269] 0506 11:39:44.091200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:44.096073 (+ 4873us) service_pool.cc:225] Handling call
0506 11:39:44.096369 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 11:39:44.096791 12881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
W20240506 11:39:44.097085 12881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.131:57337 (request call id 44) took 5 ms (client timeout 6 ms). Trace:
W20240506 11:39:44.097183 12881 rpcz_store.cc:269] 0506 11:39:44.091454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:44.096716 (+ 5262us) service_pool.cc:225] Handling call
0506 11:39:44.097071 (+ 355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 11:39:44.097404 12880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
I20240506 11:39:44.097842 12879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:44.098212 12871 raft_consensus.cc:1261] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 5 FOLLOWER]: Refusing update from remote peer 89b865272ad043919295f14ca4025cf6: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 5 index: 53. (index mismatch)
W20240506 11:39:44.098213 12879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.131:57337 (request call id 46) took 6 ms (client timeout 7 ms). Trace:
W20240506 11:39:44.098455 12879 rpcz_store.cc:269] 0506 11:39:44.091651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:44.097757 (+ 6106us) service_pool.cc:225] Handling call
0506 11:39:44.098194 (+ 437us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240506 11:39:44.098374 12878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 3 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
I20240506 11:39:44.098673 12878 raft_consensus.cc:2354] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 89b865272ad043919295f14ca4025cf6 for earlier term 3. Current term is 5.
W20240506 11:39:44.099016 12877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.131:57337 (request call id 48) took 7 ms (client timeout 7 ms). Trace:
W20240506 11:39:44.099154 12877 rpcz_store.cc:269] 0506 11:39:44.091847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:44.098865 (+ 7018us) service_pool.cc:212] Skipping call since client already timed out
0506 11:39:44.099002 (+ 137us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240506 11:39:44.099472 12884 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.130:56173 (request call id 39) took 8 ms (client timeout 6 ms). Trace:
I20240506 11:39:44.099498 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
W20240506 11:39:44.099609 12884 rpcz_store.cc:269] 0506 11:39:44.090869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:44.099392 (+ 8523us) service_pool.cc:212] Skipping call since client already timed out
0506 11:39:44.099459 (+ 67us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240506 11:39:44.099807 12876 raft_consensus.cc:2354] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89b865272ad043919295f14ca4025cf6 for earlier term 4. Current term is 5.
I20240506 11:39:44.099987 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 89b865272ad043919295f14ca4025cf6)
I20240506 11:39:44.100076 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
I20240506 11:39:44.100142 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:44.100338 12875 raft_consensus.cc:2354] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 89b865272ad043919295f14ca4025cf6 for earlier term 4. Current term is 5.
I20240506 11:39:44.100644 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 4 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab"
I20240506 11:39:44.100767 13475 leader_election.cc:288] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 89b865272ad043919295f14ca4025cf6 (127.10.45.131:32893), 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081)
I20240506 11:39:44.100878 12874 raft_consensus.cc:2354] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 89b865272ad043919295f14ca4025cf6 for earlier term 4. Current term is 5.
I20240506 11:39:44.101136 13462 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:44.101274 13462 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:44.101228 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "89b865272ad043919295f14ca4025cf6" candidate_term: 5 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" is_pre_election: true
I20240506 11:39:44.101578 12873 raft_consensus.cc:2454] T 831a913fdab244808c7b4bae4a4a56c3 P 4d74671f77b041ae90c1cb2bc6d6c5ab [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89b865272ad043919295f14ca4025cf6 in term 5.
W20240506 11:39:44.101878 12872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.131:57337 (request call id 55) took 9 ms (client timeout 8 ms). Trace:
I20240506 11:39:44.102246 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 12804
I20240506 11:39:44.102424 13017 raft_consensus.cc:1226] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 5 FOLLOWER]: Rejecting Update request from peer 4d74671f77b041ae90c1cb2bc6d6c5ab for earlier term 1. Current term is 5. Ops: []
I20240506 11:39:44.102537 13016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" candidate_term: 6 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "30ffd8145bbd483bbb01f029813c4541" is_pre_election: true
I20240506 11:39:44.102794 13016 raft_consensus.cc:2396] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d74671f77b041ae90c1cb2bc6d6c5ab for term 6 because replica has last-logged OpId of term: 5 index: 53, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 52.
I20240506 11:39:44.103129 13146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" candidate_term: 6 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
I20240506 11:39:44.103567 13147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831a913fdab244808c7b4bae4a4a56c3" candidate_uuid: "30ffd8145bbd483bbb01f029813c4541" candidate_term: 6 candidate_status { last_received { term: 5 index: 53 } } ignore_live_leader: false dest_uuid: "89b865272ad043919295f14ca4025cf6" is_pre_election: true
W20240506 11:39:44.121003 12953 leader_election.cc:334] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Timed out: RequestConsensusVote RPC to 127.10.45.129:33081 timed out after 0.006s (SENT)
I20240506 11:39:44.121176 12953 leader_election.cc:302] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30ffd8145bbd483bbb01f029813c4541; no voters: 4d74671f77b041ae90c1cb2bc6d6c5ab, 89b865272ad043919295f14ca4025cf6
I20240506 11:39:44.121456 13475 raft_consensus.cc:2726] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240506 11:39:44.137157 13080 connection.cc:535] server connection from 127.10.45.129:57675 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 11:39:44.137156 12951 connection.cc:535] server connection from 127.10.45.129:39005 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 11:39:44.137507 12953 connection.cc:535] client connection to 127.10.45.129:33081 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 11:39:44.138067 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 12935
W20240506 11:39:44.140270 13083 consensus_peers.cc:485] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 -> Peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Couldn't send request to peer 4d74671f77b041ae90c1cb2bc6d6c5ab. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:39:44.144464 13057 generic_service.cc:196] Checking for leaks (request via RPC)
W20240506 11:39:44.155666 13083 consensus_peers.cc:485] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 -> Peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Couldn't send request to peer 4d74671f77b041ae90c1cb2bc6d6c5ab. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33081: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240506 11:39:44.165594 13083 consensus_peers.cc:485] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 -> Peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Couldn't send request to peer 4d74671f77b041ae90c1cb2bc6d6c5ab. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33081: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240506 11:39:44.175753 13083 consensus_peers.cc:485] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 -> Peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Couldn't send request to peer 4d74671f77b041ae90c1cb2bc6d6c5ab. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33081: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240506 11:39:44.182478 13083 consensus_peers.cc:485] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 -> Peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Couldn't send request to peer 4d74671f77b041ae90c1cb2bc6d6c5ab. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33081: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240506 11:39:44.192442 13083 consensus_peers.cc:485] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 -> Peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Couldn't send request to peer 4d74671f77b041ae90c1cb2bc6d6c5ab. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33081: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240506 11:39:44.203536 13083 consensus_peers.cc:485] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 -> Peer 4d74671f77b041ae90c1cb2bc6d6c5ab (127.10.45.129:33081): Couldn't send request to peer 4d74671f77b041ae90c1cb2bc6d6c5ab. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33081: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240506 11:39:44.273092 13475 raft_consensus.cc:482] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 89b865272ad043919295f14ca4025cf6)
I20240506 11:39:44.273314 13475 raft_consensus.cc:504] T 831a913fdab244808c7b4bae4a4a56c3 P 30ffd8145bbd483bbb01f029813c4541 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89b865272ad043919295f14ca4025cf6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 32893 } } peers { permanent_uuid: "30ffd8145bbd483bbb01f029813c4541" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41551 } } peers { permanent_uuid: "4d74671f77b041ae90c1cb2bc6d6c5ab" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33081 } }
I20240506 11:39:44.273640 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 12935
I20240506 11:39:44.298103 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 13065
W20240506 11:39:44.298409 13083 connection.cc:535] client connection to 127.10.45.130:41551 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 11:39:44.299991 13083 consensus_peers.cc:485] T 831a913fdab244808c7b4bae4a4a56c3 P 89b865272ad043919295f14ca4025cf6 -> Peer 30ffd8145bbd483bbb01f029813c4541 (127.10.45.130:41551): Couldn't send request to peer 30ffd8145bbd483bbb01f029813c4541. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.130:41551: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:39:44.305068 13187 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:39:44.432547 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 13065
I20240506 11:39:44.457852 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 12714
I20240506 11:39:44.458861 12775 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:39:44.562022 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 12714
2024-05-06T11:39:44Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency (34937 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys
I20240506 11:39:44.603132 10422 test_util.cc:268] Using random seed: -2036558873
I20240506 11:39:44.604703 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:39:44.604758 10422 ts_itest-base.cc:113] --------------
I20240506 11:39:44.604797 10422 ts_itest-base.cc:114] 3 tablet servers
I20240506 11:39:44.604828 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:39:44.604871 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:39:44Z Disabled control of system clock
I20240506 11:39:44.630067 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:44121
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:44121 with env {}
W20240506 11:39:44.932116 13549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:39:44.932443 13549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:39:44.932569 13549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:39:44.940171 13549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:39:44.940284 13549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:39:44.940361 13549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:39:44.940446 13549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:39:44.948522 13549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:44121
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:44121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:39:44.949760 13549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:39:44.951453 13549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:39:44.961891 13554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:44.962834 13557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:44.963660 13555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:39:44.964541 13549 server_base.cc:1034] running on GCE node
I20240506 11:39:44.965279 13549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:39:44.966565 13549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:39:44.967768 13549 hybrid_clock.cc:648] HybridClock initialized: now 1714995584967710 us; error 33 us; skew 500 ppm
I20240506 11:39:44.968067 13549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:39:44.970144 13549 webserver.cc:460] Webserver started at http://127.10.45.190:37345/ using document root <none> and password file <none>
I20240506 11:39:44.970805 13549 fs_manager.cc:359] Metadata directory not provided
I20240506 11:39:44.970925 13549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:39:44.971235 13549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:39:44.973757 13549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "59610e18980643659c687eedd054f8f0"
format_stamp: "Formatted at 2024-05-06 11:39:44 on dist-test-slave-7l0g"
I20240506 11:39:44.974488 13549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "59610e18980643659c687eedd054f8f0"
format_stamp: "Formatted at 2024-05-06 11:39:44 on dist-test-slave-7l0g"
I20240506 11:39:44.979580 13549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240506 11:39:44.982765 13563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:44.984103 13549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240506 11:39:44.984257 13549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "59610e18980643659c687eedd054f8f0"
format_stamp: "Formatted at 2024-05-06 11:39:44 on dist-test-slave-7l0g"
I20240506 11:39:44.984439 13549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:39:45.006750 13549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:39:45.007656 13549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:39:45.007933 13549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:39:45.025358 13549 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:44121
I20240506 11:39:45.025388 13614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:44121 every 8 connection(s)
I20240506 11:39:45.026698 13549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:39:45.030149 13615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:39:45.035198 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 13549
I20240506 11:39:45.035389 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:39:45.040032 13615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0: Bootstrap starting.
I20240506 11:39:45.042837 13615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0: Neither blocks nor log segments found. Creating new log.
I20240506 11:39:45.043804 13615 log.cc:824] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0: Log is configured to *not* fsync() on all Append() calls
I20240506 11:39:45.046470 13615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0: No bootstrap required, opened a new log
I20240506 11:39:45.052709 13615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59610e18980643659c687eedd054f8f0" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44121 } }
I20240506 11:39:45.053014 13615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:39:45.053112 13615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59610e18980643659c687eedd054f8f0, State: Initialized, Role: FOLLOWER
I20240506 11:39:45.053653 13615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59610e18980643659c687eedd054f8f0" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44121 } }
I20240506 11:39:45.053843 13615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:39:45.053961 13615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:39:45.054108 13615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:39:45.056354 13615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59610e18980643659c687eedd054f8f0" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44121 } }
I20240506 11:39:45.056767 13615 leader_election.cc:302] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59610e18980643659c687eedd054f8f0; no voters:
I20240506 11:39:45.057296 13615 leader_election.cc:288] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:39:45.057413 13620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:39:45.058019 13620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [term 1 LEADER]: Becoming Leader. State: Replica: 59610e18980643659c687eedd054f8f0, State: Running, Role: LEADER
I20240506 11:39:45.058504 13620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59610e18980643659c687eedd054f8f0" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44121 } }
I20240506 11:39:45.059067 13615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:39:45.061084 13621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59610e18980643659c687eedd054f8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59610e18980643659c687eedd054f8f0" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44121 } } }
I20240506 11:39:45.061411 13621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [sys.catalog]: This master's current role is: LEADER
I20240506 11:39:45.061905 13622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59610e18980643659c687eedd054f8f0. Latest consensus state: current_term: 1 leader_uuid: "59610e18980643659c687eedd054f8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59610e18980643659c687eedd054f8f0" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44121 } } }
I20240506 11:39:45.062139 13622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0 [sys.catalog]: This master's current role is: LEADER
I20240506 11:39:45.065193 13630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:39:45.068994 13630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:39:45.077286 13630 catalog_manager.cc:1347] Generated new cluster ID: 14031eb10844489ca1f0863ec8bcd3fb
I20240506 11:39:45.077391 13630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:39:45.093122 13630 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:39:45.094336 13630 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:39:45.102723 13630 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 59610e18980643659c687eedd054f8f0: Generated new TSK 0
I20240506 11:39:45.103605 13630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:39:45.108351 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:44121
--builtin_ntp_servers=127.10.45.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20240506 11:39:45.413583 13639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:39:45.413854 13639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:39:45.413995 13639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:39:45.420799 13639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:39:45.420982 13639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:39:45.429180 13639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.45.190:44121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:39:45.430334 13639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:39:45.432044 13639 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:39:45.443094 13645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:45.443946 13647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:45.444766 13644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:39:45.445575 13639 server_base.cc:1034] running on GCE node
I20240506 11:39:45.446113 13639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:39:45.447013 13639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:39:45.448220 13639 hybrid_clock.cc:648] HybridClock initialized: now 1714995585448165 us; error 39 us; skew 500 ppm
I20240506 11:39:45.448498 13639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:39:45.450484 13639 webserver.cc:460] Webserver started at http://127.10.45.129:42887/ using document root <none> and password file <none>
I20240506 11:39:45.451156 13639 fs_manager.cc:359] Metadata directory not provided
I20240506 11:39:45.451273 13639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:39:45.451532 13639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:39:45.453917 13639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "37ed28a1008e4e4c930eb4a15c156b9b"
format_stamp: "Formatted at 2024-05-06 11:39:45 on dist-test-slave-7l0g"
I20240506 11:39:45.454578 13639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "37ed28a1008e4e4c930eb4a15c156b9b"
format_stamp: "Formatted at 2024-05-06 11:39:45 on dist-test-slave-7l0g"
I20240506 11:39:45.459192 13639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20240506 11:39:45.462231 13653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:45.463550 13639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240506 11:39:45.463691 13639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "37ed28a1008e4e4c930eb4a15c156b9b"
format_stamp: "Formatted at 2024-05-06 11:39:45 on dist-test-slave-7l0g"
I20240506 11:39:45.463845 13639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:39:45.494307 13639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:39:45.495105 13639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:39:45.495348 13639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:39:45.496234 13639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:39:45.497565 13639 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:39:45.497627 13639 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:45.497720 13639 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:39:45.497764 13639 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:45.526295 13639 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:33899
I20240506 11:39:45.526337 13765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:33899 every 8 connection(s)
I20240506 11:39:45.527626 13639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:39:45.537817 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 13639
I20240506 11:39:45.538074 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:39:45.539034 13766 heartbeater.cc:347] Connected to a master server at 127.10.45.190:44121
I20240506 11:39:45.539304 13766 heartbeater.cc:464] Registering TS with master...
I20240506 11:39:45.540218 13766 heartbeater.cc:511] Master 127.10.45.190:44121 requested a full tablet report, sending...
I20240506 11:39:45.541316 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:44121
--builtin_ntp_servers=127.10.45.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20240506 11:39:45.542125 13580 ts_manager.cc:194] Registered new tserver with Master: 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:45.543525 13580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:43427
W20240506 11:39:45.843123 13769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:39:45.843374 13769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:39:45.843477 13769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:39:45.850371 13769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:39:45.850531 13769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:39:45.858186 13769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.45.190:44121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:39:45.859367 13769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:39:45.860952 13769 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:39:45.872180 13774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:45.873056 13777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:45.873899 13775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:39:45.874713 13769 server_base.cc:1034] running on GCE node
I20240506 11:39:45.875264 13769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:39:45.876024 13769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:39:45.877245 13769 hybrid_clock.cc:648] HybridClock initialized: now 1714995585877183 us; error 35 us; skew 500 ppm
I20240506 11:39:45.877547 13769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:39:45.879673 13769 webserver.cc:460] Webserver started at http://127.10.45.130:36151/ using document root <none> and password file <none>
I20240506 11:39:45.880326 13769 fs_manager.cc:359] Metadata directory not provided
I20240506 11:39:45.880448 13769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:39:45.880744 13769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:39:45.883189 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d0a4d531336c4b4db79a960a8a241965"
format_stamp: "Formatted at 2024-05-06 11:39:45 on dist-test-slave-7l0g"
I20240506 11:39:45.883927 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d0a4d531336c4b4db79a960a8a241965"
format_stamp: "Formatted at 2024-05-06 11:39:45 on dist-test-slave-7l0g"
I20240506 11:39:45.888429 13769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:39:45.891531 13783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:45.892664 13769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240506 11:39:45.892804 13769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d0a4d531336c4b4db79a960a8a241965"
format_stamp: "Formatted at 2024-05-06 11:39:45 on dist-test-slave-7l0g"
I20240506 11:39:45.892956 13769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:39:45.933050 13769 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:39:45.933876 13769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:39:45.934152 13769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:39:45.935111 13769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:39:45.936528 13769 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:39:45.936606 13769 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:45.936733 13769 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:39:45.936784 13769 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:45.966135 13769 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:46563
I20240506 11:39:45.966174 13895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:46563 every 8 connection(s)
I20240506 11:39:45.967468 13769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:39:45.969202 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 13769
I20240506 11:39:45.969375 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:39:45.972611 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:44121
--builtin_ntp_servers=127.10.45.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20240506 11:39:45.980355 13896 heartbeater.cc:347] Connected to a master server at 127.10.45.190:44121
I20240506 11:39:45.980676 13896 heartbeater.cc:464] Registering TS with master...
I20240506 11:39:45.981609 13896 heartbeater.cc:511] Master 127.10.45.190:44121 requested a full tablet report, sending...
I20240506 11:39:45.986292 13580 ts_manager.cc:194] Registered new tserver with Master: d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563)
I20240506 11:39:45.987155 13580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:59011
W20240506 11:39:46.288782 13899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:39:46.289048 13899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:39:46.289181 13899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:39:46.296085 13899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:39:46.296294 13899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:39:46.304805 13899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.45.190:44121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:39:46.306007 13899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:39:46.307605 13899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:39:46.319326 13907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:46.320228 13905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:39:46.321063 13904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:39:46.321916 13899 server_base.cc:1034] running on GCE node
I20240506 11:39:46.322459 13899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:39:46.323304 13899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:39:46.324503 13899 hybrid_clock.cc:648] HybridClock initialized: now 1714995586324434 us; error 45 us; skew 500 ppm
I20240506 11:39:46.324777 13899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:39:46.326807 13899 webserver.cc:460] Webserver started at http://127.10.45.131:40951/ using document root <none> and password file <none>
I20240506 11:39:46.327512 13899 fs_manager.cc:359] Metadata directory not provided
I20240506 11:39:46.327636 13899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:39:46.327916 13899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:39:46.330296 13899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e1722c8227e641a5a4ac218853a639ab"
format_stamp: "Formatted at 2024-05-06 11:39:46 on dist-test-slave-7l0g"
I20240506 11:39:46.331008 13899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e1722c8227e641a5a4ac218853a639ab"
format_stamp: "Formatted at 2024-05-06 11:39:46 on dist-test-slave-7l0g"
I20240506 11:39:46.335608 13899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:39:46.338513 13913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:46.339733 13899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240506 11:39:46.339877 13899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e1722c8227e641a5a4ac218853a639ab"
format_stamp: "Formatted at 2024-05-06 11:39:46 on dist-test-slave-7l0g"
I20240506 11:39:46.340039 13899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:39:46.354604 13899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:39:46.355310 13899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:39:46.355557 13899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:39:46.356331 13899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:39:46.357719 13899 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:39:46.357793 13899 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:46.357918 13899 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:39:46.357972 13899 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:39:46.387642 13899 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:43815
I20240506 11:39:46.387681 14025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:43815 every 8 connection(s)
I20240506 11:39:46.388792 13899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:39:46.392208 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 13899
I20240506 11:39:46.392386 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:39:46.400003 14026 heartbeater.cc:347] Connected to a master server at 127.10.45.190:44121
I20240506 11:39:46.400230 14026 heartbeater.cc:464] Registering TS with master...
I20240506 11:39:46.400908 14026 heartbeater.cc:511] Master 127.10.45.190:44121 requested a full tablet report, sending...
I20240506 11:39:46.402046 13580 ts_manager.cc:194] Registered new tserver with Master: e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815)
I20240506 11:39:46.402763 13580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:51211
I20240506 11:39:46.407195 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:39:46.410110 10422 test_util.cc:268] Using random seed: -2034751912
I20240506 11:39:46.426216 13580 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60094:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 11:39:46.427637 13580 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 11:39:46.456224 13831 tablet_service.cc:1467] Processing CreateTablet for tablet 7d05d3dc7cfc4d31aaaa2fded12e288d (DEFAULT_TABLE table=test-workload [id=ed31e9f769364ceb898cd8f8dddef137]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:39:46.457614 13831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d05d3dc7cfc4d31aaaa2fded12e288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:39:46.459357 13961 tablet_service.cc:1467] Processing CreateTablet for tablet 7d05d3dc7cfc4d31aaaa2fded12e288d (DEFAULT_TABLE table=test-workload [id=ed31e9f769364ceb898cd8f8dddef137]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:39:46.460652 13961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d05d3dc7cfc4d31aaaa2fded12e288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:39:46.462180 13701 tablet_service.cc:1467] Processing CreateTablet for tablet 7d05d3dc7cfc4d31aaaa2fded12e288d (DEFAULT_TABLE table=test-workload [id=ed31e9f769364ceb898cd8f8dddef137]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:39:46.463693 13701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d05d3dc7cfc4d31aaaa2fded12e288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:39:46.471601 14045 tablet_bootstrap.cc:492] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965: Bootstrap starting.
I20240506 11:39:46.475492 14046 tablet_bootstrap.cc:492] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b: Bootstrap starting.
I20240506 11:39:46.475612 14044 tablet_bootstrap.cc:492] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab: Bootstrap starting.
I20240506 11:39:46.477907 14044 tablet_bootstrap.cc:654] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab: Neither blocks nor log segments found. Creating new log.
I20240506 11:39:46.477991 14046 tablet_bootstrap.cc:654] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b: Neither blocks nor log segments found. Creating new log.
I20240506 11:39:46.478909 14045 tablet_bootstrap.cc:654] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965: Neither blocks nor log segments found. Creating new log.
I20240506 11:39:46.479146 14044 log.cc:824] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab: Log is configured to *not* fsync() on all Append() calls
I20240506 11:39:46.479190 14046 log.cc:824] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b: Log is configured to *not* fsync() on all Append() calls
I20240506 11:39:46.480589 14045 log.cc:824] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965: Log is configured to *not* fsync() on all Append() calls
I20240506 11:39:46.483084 14044 tablet_bootstrap.cc:492] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab: No bootstrap required, opened a new log
I20240506 11:39:46.483223 14046 tablet_bootstrap.cc:492] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b: No bootstrap required, opened a new log
I20240506 11:39:46.483222 14045 tablet_bootstrap.cc:492] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965: No bootstrap required, opened a new log
I20240506 11:39:46.483297 14044 ts_tablet_manager.cc:1367] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240506 11:39:46.483456 14046 ts_tablet_manager.cc:1367] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240506 11:39:46.483456 14045 ts_tablet_manager.cc:1367] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20240506 11:39:46.488607 14044 raft_consensus.cc:348] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.489004 14044 raft_consensus.cc:374] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:39:46.489126 14044 raft_consensus.cc:729] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1722c8227e641a5a4ac218853a639ab, State: Initialized, Role: FOLLOWER
I20240506 11:39:46.489912 14044 consensus_queue.cc:260] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.491080 14026 heartbeater.cc:503] Master 127.10.45.190:44121 was elected leader, sending a full tablet report...
I20240506 11:39:46.491644 14046 raft_consensus.cc:348] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.491644 14045 raft_consensus.cc:348] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.491997 14046 raft_consensus.cc:374] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:39:46.491986 14045 raft_consensus.cc:374] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:39:46.492105 14046 raft_consensus.cc:729] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ed28a1008e4e4c930eb4a15c156b9b, State: Initialized, Role: FOLLOWER
I20240506 11:39:46.492105 14045 raft_consensus.cc:729] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a4d531336c4b4db79a960a8a241965, State: Initialized, Role: FOLLOWER
I20240506 11:39:46.492764 14046 consensus_queue.cc:260] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.496905 14050 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:46.497143 14050 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.497614 14044 ts_tablet_manager.cc:1398] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240506 11:39:46.509459 13766 heartbeater.cc:503] Master 127.10.45.190:44121 was elected leader, sending a full tablet report...
W20240506 11:39:46.511277 13915 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.45.130:46563 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20240506 11:39:46.513541 14054 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:46.513868 14054 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.522419 14050 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:46.528308 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:46.530617 14045 consensus_queue.cc:260] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.531136 14054 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563)
W20240506 11:39:46.532346 13656 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.45.131:43815 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
I20240506 11:39:46.533360 13851 raft_consensus.cc:2454] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1722c8227e641a5a4ac218853a639ab in term 0.
W20240506 11:39:46.533987 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563): Timed out: RequestConsensusVote RPC to 127.10.45.130:46563 timed out after 0.007s (SENT)
W20240506 11:39:46.534533 13851 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.45.131:53631 (request call id 1) took 6 ms (client timeout 7 ms). Trace:
W20240506 11:39:46.534668 13851 rpcz_store.cc:269] 0506 11:39:46.527731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:46.527802 (+ 71us) service_pool.cc:225] Handling call
0506 11:39:46.534369 (+ 6567us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x616000081454. stack: 00007fbc1919a756 00007fbc1919a510 00007fbc181cd48d 0000000000955804 00000000009557cd 000000000095519a 00007fbc249f96b5 00007fbc286cbb04 00007fbc22dcf9b6 00007fbc22dcf6e5 00007fbc1fea204e 00007fbc1fea0f2e 00007fbc1fea7405 00007fbc1fea7d2f 00007fbc1fea7b58 0000000000a60bcc
0506 11:39:46.534455 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10178304}
I20240506 11:39:46.536106 14052 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:39:46.536278 14052 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.538646 14052 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:46.541860 14046 ts_tablet_manager.cc:1398] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b: Time spent starting tablet: real 0.058s user 0.032s sys 0.025s
W20240506 11:39:46.543007 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Timed out: connection negotiation to 127.10.45.129:33899 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20240506 11:39:46.543165 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:46.545850 14051 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240506 11:39:46.548166 13896 heartbeater.cc:503] Master 127.10.45.190:44121 was elected leader, sending a full tablet report...
I20240506 11:39:46.548784 14045 ts_tablet_manager.cc:1398] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965: Time spent starting tablet: real 0.065s user 0.011s sys 0.006s
I20240506 11:39:46.550588 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:46.551120 13981 raft_consensus.cc:2454] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ed28a1008e4e4c930eb4a15c156b9b in term 0.
W20240506 11:39:46.551642 13786 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.45.131:43815 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
I20240506 11:39:46.551735 13656 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab; no voters:
I20240506 11:39:46.553718 14053 raft_consensus.cc:2781] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 11:39:46.553953 14053 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:46.555083 14053 raft_consensus.cc:3037] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:39:46.559418 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:46.559607 13981 raft_consensus.cc:2454] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a4d531336c4b4db79a960a8a241965 in term 0.
I20240506 11:39:46.560173 13786 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965, e1722c8227e641a5a4ac218853a639ab; no voters:
I20240506 11:39:46.560580 14052 raft_consensus.cc:2781] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 11:39:46.560731 14052 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:39:46.560845 14052 raft_consensus.cc:3037] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:39:46.560269 14053 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.561488 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:46.562489 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965"
I20240506 11:39:46.562821 14053 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [CANDIDATE]: Term 1 election: Requested vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563)
I20240506 11:39:46.562796 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:46.563246 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab"
I20240506 11:39:46.563387 13721 raft_consensus.cc:2379] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0a4d531336c4b4db79a960a8a241965 in current term 1: Already voted for candidate 37ed28a1008e4e4c930eb4a15c156b9b in this term.
I20240506 11:39:46.563421 13981 raft_consensus.cc:3037] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:39:46.564554 14052 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.564910 13851 raft_consensus.cc:2379] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37ed28a1008e4e4c930eb4a15c156b9b in current term 1: Already voted for candidate d0a4d531336c4b4db79a960a8a241965 in this term.
W20240506 11:39:46.564985 13655 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563): Timed out: RequestConsensusVote RPC to 127.10.45.130:46563 timed out after 0.008s (SENT)
I20240506 11:39:46.565173 14052 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 1 election: Requested vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:46.565698 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b"
I20240506 11:39:46.565910 13721 raft_consensus.cc:2379] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a4d531336c4b4db79a960a8a241965 in current term 1: Already voted for candidate 37ed28a1008e4e4c930eb4a15c156b9b in this term.
I20240506 11:39:46.565852 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab"
I20240506 11:39:46.565979 13981 raft_consensus.cc:2454] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ed28a1008e4e4c930eb4a15c156b9b in term 1.
I20240506 11:39:46.566323 13656 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab; no voters: d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:46.566567 14053 raft_consensus.cc:2781] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:39:46.566641 13786 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:46.566723 14053 raft_consensus.cc:686] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [term 1 LEADER]: Becoming Leader. State: Replica: 37ed28a1008e4e4c930eb4a15c156b9b, State: Running, Role: LEADER
I20240506 11:39:46.566900 14052 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240506 11:39:46.567212 14053 consensus_queue.cc:237] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.571215 14053 consensus_queue.cc:1035] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:39:46.572026 14053 consensus_queue.cc:1035] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:39:46.571753 13580 catalog_manager.cc:5484] T 7d05d3dc7cfc4d31aaaa2fded12e288d P 37ed28a1008e4e4c930eb4a15c156b9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129). New cstate: current_term: 1 leader_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } health_report { overall_health: HEALTHY } } }
I20240506 11:39:46.584580 14052 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:46.584756 14052 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:46.586040 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:46.586405 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:46.587110 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:46.589239 14052 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:46.594722 14071 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 11:39:46.613617 14067 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240506 11:39:46.613696 14067 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20240506 11:39:47.020141 13851 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240506 11:39:47.020156 13980 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240506 11:39:47.020226 13851 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20240506 11:39:47.020241 13980 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240506 11:39:47.482318 14071 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:47.482578 14071 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:47.483428 14071 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:47.484069 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:47.484149 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:47.485266 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:47.485930 14071 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:47.840545 14051 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:47.840766 14051 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:47.841991 14051 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:47.842612 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:47.842839 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:47.843645 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:47.844007 14051 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:48.270963 14071 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:48.271195 14071 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:48.271943 14071 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:48.272482 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:48.272622 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:48.273294 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:48.273798 14083 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=1, attempt_no=0}) took 1665 ms. Trace:
I20240506 11:39:48.273958 14083 rpcz_store.cc:276] 0506 11:39:46.608273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:46.608330 (+ 57us) service_pool.cc:225] Handling call
0506 11:39:48.273733 (+1665403us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:47.018605 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:47.018739 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:47.018747 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:39:47.018749 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:47.018844 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:47.018891 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:47.018896 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:47.018974 (+ 78us) tablet.cc:699] Row locks acquired
0506 11:39:47.018979 (+ 5us) write_op.cc:260] PREPARE: finished
0506 11:39:47.019034 (+ 55us) write_op.cc:270] Start()
0506 11:39:47.665173 (+646139us) write_op.cc:276] Timestamp: P: 1714995587019007 usec, L: 0
0506 11:39:47.665186 (+ 13us) op_driver.cc:331] REPLICATION: starting
0506 11:39:47.665393 (+ 207us) log.cc:842] Serialized 345 byte log entry
0506 11:39:48.272714 (+607321us) op_driver.cc:444] REPLICATION: finished
0506 11:39:48.273172 (+ 458us) write_op.cc:301] APPLY: starting
0506 11:39:48.273191 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:48.273233 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:48.273279 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:48.273377 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:48.273429 (+ 52us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:48.273439 (+ 10us) write_op.cc:312] APPLY: finished
0506 11:39:48.273532 (+ 93us) log.cc:842] Serialized 22 byte log entry
0506 11:39:48.273626 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:48.273645 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:48.273652 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":364544,"child_traces":[["op",{"apply.queue_time_us":373,"mutex_wait_us":34,"num_ops":1,"prepare.queue_time_us":406825,"prepare.run_cpu_time_us":798,"prepare.run_wall_time_us":646870,"replication_time_us":607502,"thread_start_us":286,"threads_started":1}]]}
I20240506 11:39:49.197382 14071 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:49.201035 14090 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=2, attempt_no=0}) took 1713 ms. Trace:
I20240506 11:39:49.201170 14090 rpcz_store.cc:276] 0506 11:39:47.487346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:47.487395 (+ 49us) service_pool.cc:225] Handling call
0506 11:39:49.200975 (+1713580us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:47.665524 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:47.665622 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:47.665627 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:47.665628 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:47.665715 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:47.665727 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:47.665731 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:47.665783 (+ 52us) tablet.cc:699] Row locks acquired
0506 11:39:47.665785 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:47.665843 (+ 58us) write_op.cc:270] Start()
0506 11:39:47.702019 (+ 36176us) write_op.cc:276] Timestamp: P: 1714995587665791 usec, L: 0
0506 11:39:47.702024 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:47.702252 (+ 228us) log.cc:842] Serialized 354 byte log entry
0506 11:39:49.198993 (+1496741us) op_driver.cc:444] REPLICATION: finished
0506 11:39:49.199401 (+ 408us) write_op.cc:301] APPLY: starting
0506 11:39:49.199418 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:49.199452 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:49.199487 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:49.199598 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:49.199623 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:49.199628 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:49.199818 (+ 190us) log.cc:842] Serialized 43 byte log entry
0506 11:39:49.199876 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:49.199903 (+ 27us) write_op.cc:454] Released schema lock
0506 11:39:49.199908 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":340,"num_ops":1,"prepare.queue_time_us":177591,"prepare.run_cpu_time_us":733,"prepare.run_wall_time_us":36808,"replication_time_us":1496939,"thread_start_us":238,"threads_started":1}]]}
I20240506 11:39:49.286229 14051 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:49.286465 14051 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:49.286811 13980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:48429 (request call id 228) took 1442 ms. Trace:
I20240506 11:39:49.287164 14051 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:49.287707 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:49.287827 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:49.286921 13980 rpcz_store.cc:276] 0506 11:39:47.844336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:47.844400 (+ 64us) service_pool.cc:225] Handling call
0506 11:39:47.844606 (+ 206us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 11:39:47.844735 (+ 129us) raft_consensus.cc:1520] Early marking committed up to index 2
0506 11:39:47.845135 (+ 400us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 11:39:47.845253 (+ 118us) write_op.cc:270] Start()
0506 11:39:48.421409 (+576156us) write_op.cc:276] Timestamp: P: 1714995587019007 usec, L: 0
0506 11:39:48.422077 (+ 668us) write_op.cc:270] Start()
0506 11:39:49.285228 (+863151us) write_op.cc:276] Timestamp: P: 1714995587665791 usec, L: 0
0506 11:39:49.285774 (+ 546us) log.cc:842] Serialized 699 byte log entry
0506 11:39:49.286110 (+ 336us) raft_consensus.cc:1631] Marking committed up to 2
0506 11:39:49.286118 (+ 8us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:49.286153 (+ 35us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:49.286672 (+ 519us) raft_consensus.cc:1671] finished
0506 11:39:49.286674 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:49.286739 (+ 65us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:48.421939 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:48.422061 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:48.422066 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:48.422067 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:48.422142 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:48.422156 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:48.422158 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:48.422204 (+ 46us) tablet.cc:699] Row locks acquired
0506 11:39:48.422205 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0506 11:39:49.285674 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:49.285800 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:49.285805 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:49.285806 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:49.285890 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:49.285919 (+ 29us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:49.285922 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:49.285974 (+ 52us) tablet.cc:699] Row locks acquired
0506 11:39:49.285976 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":242,"threads_started":1,"wal-append.queue_time_us":324,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":392,"prepare.run_cpu_time_us":340,"prepare.run_wall_time_us":340,"thread_start_us":279,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":23,"prepare.queue_time_us":323,"prepare.run_cpu_time_us":490,"prepare.run_wall_time_us":490,"thread_start_us":205,"threads_started":1}]]}
I20240506 11:39:49.288357 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:49.288669 14051 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:49.821244 14071 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:49.821473 14071 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:49.822341 14071 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:49.822841 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:49.823035 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:49.823709 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:49.824033 14071 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:49.824566 14100 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=3, attempt_no=0}) took 1548 ms. Trace:
I20240506 11:39:49.824695 14100 rpcz_store.cc:276] 0506 11:39:48.276502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:48.276599 (+ 97us) service_pool.cc:225] Handling call
0506 11:39:49.824508 (+1547909us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:48.277264 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:48.277402 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:48.277407 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:48.277410 (+ 3us) tablet.cc:660] Decoding operations
0506 11:39:48.277516 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:48.277530 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:48.277535 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:48.277620 (+ 85us) tablet.cc:699] Row locks acquired
0506 11:39:48.277621 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:48.277654 (+ 33us) write_op.cc:270] Start()
0506 11:39:48.563029 (+285375us) write_op.cc:276] Timestamp: P: 1714995588277622 usec, L: 0
0506 11:39:48.563048 (+ 19us) op_driver.cc:331] REPLICATION: starting
0506 11:39:48.563311 (+ 263us) log.cc:842] Serialized 354 byte log entry
0506 11:39:49.823604 (+1260293us) op_driver.cc:444] REPLICATION: finished
0506 11:39:49.824009 (+ 405us) write_op.cc:301] APPLY: starting
0506 11:39:49.824025 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:49.824074 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:49.824107 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:49.824191 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:49.824237 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:49.824243 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:39:49.824323 (+ 80us) log.cc:842] Serialized 22 byte log entry
0506 11:39:49.824395 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:49.824413 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:49.824420 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":337,"num_ops":1,"prepare.queue_time_us":282,"prepare.run_cpu_time_us":881,"prepare.run_wall_time_us":286120,"replication_time_us":1260530,"thread_start_us":418,"threads_started":2}]]}
I20240506 11:39:49.941612 14071 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:49.941849 14071 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:49.942598 14071 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:49.943254 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:49.943468 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:49.944367 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:49.944659 14071 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:50.150620 14051 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:50.150863 14051 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:50.151638 14051 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:50.151907 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:50.152128 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:50.152698 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:50.345839 14051 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:51.075682 14104 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:51.075868 14104 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:51.076638 14104 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:51.077217 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:51.077253 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:51.078028 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:51.078413 14107 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=5, attempt_no=0}) took 1252 ms. Trace:
I20240506 11:39:51.078563 14104 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:51.078517 14107 rpcz_store.cc:276] 0506 11:39:49.826248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:49.826364 (+ 116us) service_pool.cc:225] Handling call
0506 11:39:51.078364 (+1252000us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:49.826820 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:49.826907 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:49.826910 (+ 3us) write_op.cc:435] Acquired schema lock
0506 11:39:49.826912 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:49.827015 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:49.827023 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:49.827026 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:49.827067 (+ 41us) tablet.cc:699] Row locks acquired
0506 11:39:49.827069 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:49.827102 (+ 33us) write_op.cc:270] Start()
0506 11:39:50.789238 (+962136us) write_op.cc:276] Timestamp: P: 1714995589827075 usec, L: 0
0506 11:39:50.789242 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:50.789409 (+ 167us) log.cc:842] Serialized 354 byte log entry
0506 11:39:51.076852 (+287443us) op_driver.cc:444] REPLICATION: finished
0506 11:39:51.077266 (+ 414us) write_op.cc:301] APPLY: starting
0506 11:39:51.077283 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:51.077327 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:51.077370 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:51.077472 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:51.077503 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:51.077510 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:39:51.077644 (+ 134us) log.cc:842] Serialized 22 byte log entry
0506 11:39:51.077730 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:51.077755 (+ 25us) write_op.cc:454] Released schema lock
0506 11:39:51.077762 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":334,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":602,"prepare.run_wall_time_us":962676,"raft.queue_time_us":81,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":243,"replication_time_us":287583,"thread_start_us":219,"threads_started":1}]]}
I20240506 11:39:51.571195 14051 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:51.571368 14051 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:51.572129 14051 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:51.572623 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:51.572721 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:51.573284 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:51.834424 14051 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:51.836668 14113 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=6, attempt_no=0}) took 1891 ms. Trace:
I20240506 11:39:51.836807 14113 rpcz_store.cc:276] 0506 11:39:49.945057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:49.945137 (+ 80us) service_pool.cc:225] Handling call
0506 11:39:51.836613 (+1891476us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:50.789534 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:50.789665 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:50.789669 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:50.789670 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:50.789777 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:50.789788 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:50.789792 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:50.789842 (+ 50us) tablet.cc:699] Row locks acquired
0506 11:39:50.789844 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:50.789880 (+ 36us) write_op.cc:270] Start()
0506 11:39:51.247009 (+457129us) write_op.cc:276] Timestamp: P: 1714995590789850 usec, L: 0
0506 11:39:51.247013 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:51.247240 (+ 227us) log.cc:842] Serialized 354 byte log entry
0506 11:39:51.835632 (+588392us) op_driver.cc:444] REPLICATION: finished
0506 11:39:51.835988 (+ 356us) write_op.cc:301] APPLY: starting
0506 11:39:51.836007 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:51.836053 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:51.836115 (+ 62us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:51.836185 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:51.836226 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:51.836232 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:39:51.836308 (+ 76us) log.cc:842] Serialized 22 byte log entry
0506 11:39:51.836386 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:51.836404 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:51.836410 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":290,"num_ops":1,"prepare.queue_time_us":844006,"prepare.run_cpu_time_us":725,"prepare.run_wall_time_us":457788,"replication_time_us":588595,"thread_start_us":221,"threads_started":1}]]}
I20240506 11:39:51.927644 14051 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:51.927891 14051 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:51.928711 14051 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:51.929316 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:51.929409 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:51.929970 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:51.930300 14051 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:51.963583 14104 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:51.963769 14104 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:51.964671 14104 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:51.965201 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:51.965314 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:51.965893 13786 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:52.006006 14104 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:53.391134 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:53.391485 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:53.392922 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:53.393841 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:53.394243 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:53.394821 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:53.395216 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:53.414891 14118 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:53.415096 14118 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:53.415830 14118 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:53.416277 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:53.416676 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:53.417191 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:53.417608 14124 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=8, attempt_no=0}) took 1579 ms. Trace:
I20240506 11:39:53.417765 14124 rpcz_store.cc:276] 0506 11:39:51.838356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:51.838420 (+ 64us) service_pool.cc:225] Handling call
0506 11:39:53.417558 (+1579138us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:51.838833 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:51.838916 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:51.838920 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:51.838921 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:51.839002 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:51.839010 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:51.839013 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:51.839047 (+ 34us) tablet.cc:699] Row locks acquired
0506 11:39:51.839049 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:51.839078 (+ 29us) write_op.cc:270] Start()
0506 11:39:52.732210 (+893132us) write_op.cc:276] Timestamp: P: 1714995591839047 usec, L: 0
0506 11:39:52.732214 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:52.732389 (+ 175us) log.cc:842] Serialized 354 byte log entry
0506 11:39:53.399825 (+667436us) op_driver.cc:444] REPLICATION: finished
0506 11:39:53.416820 (+ 16995us) write_op.cc:301] APPLY: starting
0506 11:39:53.416838 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:53.416878 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:53.416907 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:53.417181 (+ 274us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:53.417228 (+ 47us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:53.417235 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:39:53.417332 (+ 97us) log.cc:842] Serialized 43 byte log entry
0506 11:39:53.417388 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:53.417403 (+ 15us) write_op.cc:454] Released schema lock
0506 11:39:53.417409 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":16775,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":927,"prepare.run_wall_time_us":894416,"raft.queue_time_us":85,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":203,"replication_time_us":667569,"thread_start_us":17849,"threads_started":2}]]}
I20240506 11:39:53.766803 14118 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:53.768335 14124 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=9, attempt_no=0}) took 1837 ms. Trace:
I20240506 11:39:53.768464 14124 rpcz_store.cc:276] 0506 11:39:51.930831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:51.930884 (+ 53us) service_pool.cc:225] Handling call
0506 11:39:53.768275 (+1837391us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:52.733266 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:52.733352 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:52.733357 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:52.733358 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:52.733424 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:52.733433 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:52.733435 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:52.733475 (+ 40us) tablet.cc:699] Row locks acquired
0506 11:39:52.733476 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:52.733510 (+ 34us) write_op.cc:270] Start()
0506 11:39:53.112653 (+379143us) write_op.cc:276] Timestamp: P: 1714995592733489 usec, L: 0
0506 11:39:53.112659 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:39:53.112867 (+ 208us) log.cc:842] Serialized 354 byte log entry
0506 11:39:53.767737 (+654870us) op_driver.cc:444] REPLICATION: finished
0506 11:39:53.767866 (+ 129us) write_op.cc:301] APPLY: starting
0506 11:39:53.767889 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:53.767931 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:53.767963 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:53.768026 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:53.768053 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:53.768058 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:53.768122 (+ 64us) log.cc:842] Serialized 22 byte log entry
0506 11:39:53.768187 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:53.768201 (+ 14us) write_op.cc:454] Released schema lock
0506 11:39:53.768206 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"num_ops":1,"prepare.queue_time_us":801970,"prepare.run_cpu_time_us":608,"prepare.run_wall_time_us":379670,"replication_time_us":655054}]]}
I20240506 11:39:54.184917 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:54.185123 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:54.185905 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:54.186262 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:54.186684 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:54.187374 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:54.187629 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:54.432960 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:54.433179 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:54.434087 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:54.434618 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:54.434768 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:54.435334 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:54.435659 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:54.436035 14132 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=10, attempt_no=0}) took 1016 ms. Trace:
I20240506 11:39:54.436199 14132 rpcz_store.cc:276] 0506 11:39:53.419588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:53.419667 (+ 79us) service_pool.cc:225] Handling call
0506 11:39:54.435959 (+1016292us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:53.420067 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:53.420156 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:53.420160 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:53.420161 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:53.420224 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:53.420232 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:53.420234 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:53.420268 (+ 34us) tablet.cc:699] Row locks acquired
0506 11:39:53.420269 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:53.420298 (+ 29us) write_op.cc:270] Start()
0506 11:39:54.276439 (+856141us) write_op.cc:276] Timestamp: P: 1714995593420272 usec, L: 0
0506 11:39:54.276443 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:54.276657 (+ 214us) log.cc:842] Serialized 354 byte log entry
0506 11:39:54.434001 (+157344us) op_driver.cc:444] REPLICATION: finished
0506 11:39:54.435151 (+ 1150us) write_op.cc:301] APPLY: starting
0506 11:39:54.435194 (+ 43us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:54.435249 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:54.435291 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:54.435378 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:54.435424 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:54.435437 (+ 13us) write_op.cc:312] APPLY: finished
0506 11:39:54.435520 (+ 83us) log.cc:842] Serialized 22 byte log entry
0506 11:39:54.435826 (+ 306us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:54.435845 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:54.435854 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1068,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":856685,"raft.queue_time_us":134,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":225,"replication_time_us":157529,"spinlock_wait_cycles":7552,"thread_start_us":956,"threads_started":1}]]}
I20240506 11:39:54.762810 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:54.763025 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:54.763834 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:54.764178 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:54.764458 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:54.765123 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:54.765395 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:55.235123 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:55.235407 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:55.236225 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:55.236698 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:55.236830 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:55.237452 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:55.237823 14137 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=11, attempt_no=0}) took 1467 ms. Trace:
I20240506 11:39:55.237933 14137 rpcz_store.cc:276] 0506 11:39:53.770305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:53.770379 (+ 74us) service_pool.cc:225] Handling call
0506 11:39:55.237761 (+1467382us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:54.276795 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:54.276912 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:54.276919 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:39:54.276923 (+ 4us) tablet.cc:660] Decoding operations
0506 11:39:54.277041 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:54.277052 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:54.277056 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:54.277111 (+ 55us) tablet.cc:699] Row locks acquired
0506 11:39:54.277113 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:54.277158 (+ 45us) write_op.cc:270] Start()
0506 11:39:54.767290 (+490132us) write_op.cc:276] Timestamp: P: 1714995594277123 usec, L: 0
0506 11:39:54.767296 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:39:54.767486 (+ 190us) log.cc:842] Serialized 354 byte log entry
0506 11:39:55.236257 (+468771us) op_driver.cc:444] REPLICATION: finished
0506 11:39:55.236779 (+ 522us) write_op.cc:301] APPLY: starting
0506 11:39:55.236796 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:55.236839 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:55.236863 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:55.236931 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:55.236960 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:55.236966 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:39:55.237120 (+ 154us) log.cc:842] Serialized 43 byte log entry
0506 11:39:55.237186 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:55.237204 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:55.237212 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":426,"num_ops":1,"prepare.queue_time_us":505996,"prepare.run_cpu_time_us":683,"prepare.run_wall_time_us":490751,"replication_time_us":468935,"thread_start_us":381,"threads_started":1}]]}
I20240506 11:39:55.570164 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:55.571802 14137 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=12, attempt_no=0}) took 1133 ms. Trace:
I20240506 11:39:55.571950 14137 rpcz_store.cc:276] 0506 11:39:54.437988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:54.438066 (+ 78us) service_pool.cc:225] Handling call
0506 11:39:55.571736 (+1133670us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:54.767576 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:54.767679 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:54.767683 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:54.767685 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:54.767761 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:54.767771 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:54.767775 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:54.767819 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:39:54.767821 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:54.767864 (+ 43us) write_op.cc:270] Start()
0506 11:39:55.009033 (+241169us) write_op.cc:276] Timestamp: P: 1714995594767832 usec, L: 0
0506 11:39:55.009038 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:55.009261 (+ 223us) log.cc:842] Serialized 354 byte log entry
0506 11:39:55.571026 (+561765us) op_driver.cc:444] REPLICATION: finished
0506 11:39:55.571136 (+ 110us) write_op.cc:301] APPLY: starting
0506 11:39:55.571153 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:55.571192 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:55.571307 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:55.571392 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:55.571426 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:55.571437 (+ 11us) write_op.cc:312] APPLY: finished
0506 11:39:55.571525 (+ 88us) log.cc:842] Serialized 43 byte log entry
0506 11:39:55.571578 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:55.571615 (+ 37us) write_op.cc:454] Released schema lock
0506 11:39:55.571622 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"num_ops":1,"prepare.queue_time_us":329058,"prepare.run_cpu_time_us":666,"prepare.run_wall_time_us":241747,"replication_time_us":561940}]]}
I20240506 11:39:55.616075 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:55.616267 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:55.617058 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:55.617380 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:55.617513 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:55.618248 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:56.171911 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:56.476358 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:56.476562 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:56.477252 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:56.477702 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:56.477705 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:56.478368 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:56.478650 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:56.479053 14144 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=13, attempt_no=0}) took 1239 ms. Trace:
I20240506 11:39:56.479163 14144 rpcz_store.cc:276] 0506 11:39:55.239637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:55.239715 (+ 78us) service_pool.cc:225] Handling call
0506 11:39:56.479009 (+1239294us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:55.240171 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:55.240258 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:55.240262 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:55.240263 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:55.240327 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:55.240335 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:55.240337 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:55.240372 (+ 35us) tablet.cc:699] Row locks acquired
0506 11:39:55.240374 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:55.240403 (+ 29us) write_op.cc:270] Start()
0506 11:39:55.773545 (+533142us) write_op.cc:276] Timestamp: P: 1714995595240375 usec, L: 0
0506 11:39:55.773549 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:55.773719 (+ 170us) log.cc:842] Serialized 354 byte log entry
0506 11:39:56.477877 (+704158us) op_driver.cc:444] REPLICATION: finished
0506 11:39:56.478480 (+ 603us) write_op.cc:301] APPLY: starting
0506 11:39:56.478498 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:56.478550 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:56.478581 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:56.478676 (+ 95us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:56.478712 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:56.478719 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:39:56.478796 (+ 77us) log.cc:842] Serialized 22 byte log entry
0506 11:39:56.478877 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:56.478895 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:56.478902 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":523,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":586,"prepare.run_wall_time_us":533663,"raft.queue_time_us":51,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":177,"replication_time_us":704299,"thread_start_us":407,"threads_started":1}]]}
I20240506 11:39:56.560616 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:56.560798 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:56.561581 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:56.562129 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:56.562134 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:56.562750 13786 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:56.603458 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:56.603679 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:56.604492 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:56.605018 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:56.605268 14144 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=14, attempt_no=0}) took 1031 ms. Trace:
I20240506 11:39:56.605297 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:56.605399 14144 rpcz_store.cc:276] 0506 11:39:55.573650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:55.573723 (+ 73us) service_pool.cc:225] Handling call
0506 11:39:56.605198 (+1031475us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:55.773866 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:55.773958 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:55.773962 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:55.773964 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:55.774037 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:55.774050 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:55.774054 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:55.774116 (+ 62us) tablet.cc:699] Row locks acquired
0506 11:39:55.774120 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:39:55.774169 (+ 49us) write_op.cc:270] Start()
0506 11:39:56.309312 (+535143us) write_op.cc:276] Timestamp: P: 1714995595774145 usec, L: 0
0506 11:39:56.309317 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:56.309554 (+ 237us) log.cc:842] Serialized 354 byte log entry
0506 11:39:56.604545 (+294991us) op_driver.cc:444] REPLICATION: finished
0506 11:39:56.604649 (+ 104us) write_op.cc:301] APPLY: starting
0506 11:39:56.604666 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:56.604717 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:56.604749 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:56.604820 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:56.604856 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:56.604863 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:39:56.604976 (+ 113us) log.cc:842] Serialized 43 byte log entry
0506 11:39:56.605040 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:56.605079 (+ 39us) write_op.cc:454] Released schema lock
0506 11:39:56.605085 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"num_ops":1,"prepare.queue_time_us":199769,"prepare.run_cpu_time_us":697,"prepare.run_wall_time_us":535752,"replication_time_us":295196}]]}
I20240506 11:39:56.605898 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:57.549582 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:57.552193 14151 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=15, attempt_no=0}) took 1071 ms. Trace:
I20240506 11:39:57.552333 14151 rpcz_store.cc:276] 0506 11:39:56.481203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:56.481286 (+ 83us) service_pool.cc:225] Handling call
0506 11:39:57.552140 (+1070854us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:56.481681 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:56.481760 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:56.481763 (+ 3us) write_op.cc:435] Acquired schema lock
0506 11:39:56.481765 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:56.481824 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:56.481831 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:56.481833 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:56.481885 (+ 52us) tablet.cc:699] Row locks acquired
0506 11:39:56.481886 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:56.481927 (+ 41us) write_op.cc:270] Start()
0506 11:39:56.530127 (+ 48200us) write_op.cc:276] Timestamp: P: 1714995596481892 usec, L: 0
0506 11:39:56.530132 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:56.530366 (+ 234us) log.cc:842] Serialized 354 byte log entry
0506 11:39:57.551172 (+1020806us) op_driver.cc:444] REPLICATION: finished
0506 11:39:57.551611 (+ 439us) write_op.cc:301] APPLY: starting
0506 11:39:57.551630 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:57.551672 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:57.551726 (+ 54us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:57.551804 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:57.551841 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:57.551848 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:39:57.551933 (+ 85us) log.cc:842] Serialized 22 byte log entry
0506 11:39:57.552021 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:57.552039 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:57.552046 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":363,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":648,"prepare.run_wall_time_us":48745,"replication_time_us":1021013,"thread_start_us":257,"threads_started":1}]]}
I20240506 11:39:57.989254 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:57.989470 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:57.990288 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:57.990787 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:57.991354 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:57.991879 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:57.992583 14151 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=16, attempt_no=0}) took 1385 ms. Trace:
I20240506 11:39:57.992700 14151 rpcz_store.cc:276] 0506 11:39:56.607310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:56.607375 (+ 65us) service_pool.cc:225] Handling call
0506 11:39:57.992524 (+1385149us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:56.607774 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:56.607859 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:56.607866 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:39:56.607870 (+ 4us) tablet.cc:660] Decoding operations
0506 11:39:56.607950 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:56.607960 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:56.607966 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:56.608010 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:39:56.608011 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:56.608044 (+ 33us) write_op.cc:270] Start()
0506 11:39:57.457208 (+849164us) write_op.cc:276] Timestamp: P: 1714995596608001 usec, L: 0
0506 11:39:57.457213 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:57.457426 (+ 213us) log.cc:842] Serialized 354 byte log entry
0506 11:39:57.991923 (+534497us) op_driver.cc:444] REPLICATION: finished
0506 11:39:57.992051 (+ 128us) write_op.cc:301] APPLY: starting
0506 11:39:57.992070 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:57.992109 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:57.992136 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:57.992194 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:57.992217 (+ 23us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:57.992222 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:39:57.992304 (+ 82us) log.cc:842] Serialized 43 byte log entry
0506 11:39:57.992365 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:57.992383 (+ 18us) write_op.cc:454] Released schema lock
0506 11:39:57.992391 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":75,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":640,"prepare.run_wall_time_us":849715,"replication_time_us":534682}]]}
I20240506 11:39:57.993399 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:58.044401 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:58.044867 13851 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:49185 (request call id 1368) took 1482 ms. Trace:
I20240506 11:39:58.044984 13851 rpcz_store.cc:276] 0506 11:39:56.562138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:56.562214 (+ 76us) service_pool.cc:225] Handling call
0506 11:39:56.562410 (+ 196us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 11:39:56.562601 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 15
0506 11:39:56.562967 (+ 366us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 11:39:56.563112 (+ 145us) write_op.cc:270] Start()
0506 11:39:57.503282 (+940170us) write_op.cc:276] Timestamp: P: 1714995595774145 usec, L: 0
0506 11:39:57.504016 (+ 734us) write_op.cc:270] Start()
0506 11:39:58.043212 (+539196us) write_op.cc:276] Timestamp: P: 1714995596481892 usec, L: 0
0506 11:39:58.043861 (+ 649us) log.cc:842] Serialized 708 byte log entry
0506 11:39:58.044252 (+ 391us) raft_consensus.cc:1631] Marking committed up to 15
0506 11:39:58.044260 (+ 8us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:58.044292 (+ 32us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:58.044754 (+ 462us) raft_consensus.cc:1671] finished
0506 11:39:58.044756 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:58.044804 (+ 48us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:57.503860 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:57.503982 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:57.503987 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:57.503989 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:57.504083 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:57.504095 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:57.504098 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:57.504156 (+ 58us) tablet.cc:699] Row locks acquired
0506 11:39:57.504157 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0506 11:39:58.043689 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:58.043815 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:58.043821 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:39:58.043823 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:58.043917 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:58.044058 (+ 141us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:58.044067 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:58.044149 (+ 82us) tablet.cc:699] Row locks acquired
0506 11:39:58.044154 (+ 5us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":316,"threads_started":1,"wal-append.queue_time_us":405,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":427,"prepare.run_cpu_time_us":403,"prepare.run_wall_time_us":403,"thread_start_us":292,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":33,"prepare.queue_time_us":336,"prepare.run_cpu_time_us":497,"prepare.run_wall_time_us":571,"thread_start_us":229,"threads_started":1}]]}
I20240506 11:39:58.340507 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:58.340694 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:58.341351 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:58.341809 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:58.342322 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:58.342782 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:58.585716 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:58.585911 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:58.586659 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:58.587121 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:39:58.587396 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:58.587936 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:39:58.625814 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:59.344588 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:39:59.344784 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:39:59.345501 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:39:59.345995 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:39:59.346319 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:39:59.346900 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:39:59.347214 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:59.347216 14160 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=19, attempt_no=0}) took 1002 ms. Trace:
I20240506 11:39:59.347316 14160 rpcz_store.cc:276] 0506 11:39:58.343617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:58.343691 (+ 74us) service_pool.cc:225] Handling call
0506 11:39:59.346525 (+1002834us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:58.344102 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:58.344175 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:58.344179 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:58.344180 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:58.344238 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:58.344244 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:58.344247 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:58.344279 (+ 32us) tablet.cc:699] Row locks acquired
0506 11:39:58.344280 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:39:58.344307 (+ 27us) write_op.cc:270] Start()
0506 11:39:59.007469 (+663162us) write_op.cc:276] Timestamp: P: 1714995598344291 usec, L: 0
0506 11:39:59.007474 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:59.007737 (+ 263us) log.cc:842] Serialized 354 byte log entry
0506 11:39:59.345542 (+337805us) op_driver.cc:444] REPLICATION: finished
0506 11:39:59.345951 (+ 409us) write_op.cc:301] APPLY: starting
0506 11:39:59.345985 (+ 34us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:59.346026 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:59.346081 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:59.346143 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:59.346178 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:59.346184 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:39:59.346312 (+ 128us) log.cc:842] Serialized 43 byte log entry
0506 11:39:59.346383 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:59.346402 (+ 19us) write_op.cc:454] Released schema lock
0506 11:39:59.346410 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":351,"num_ops":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":644,"prepare.run_wall_time_us":663733,"raft.queue_time_us":43,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":338041,"thread_start_us":295,"threads_started":1}]]}
I20240506 11:39:59.758771 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:39:59.759195 13851 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:49185 (request call id 1370) took 1171 ms. Trace:
I20240506 11:39:59.759299 13851 rpcz_store.cc:276] 0506 11:39:58.587278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:58.587336 (+ 58us) service_pool.cc:225] Handling call
0506 11:39:58.587613 (+ 277us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 11:39:58.587765 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 11:39:58.587769 (+ 4us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 11:39:58.587879 (+ 110us) write_op.cc:270] Start()
0506 11:39:59.348014 (+760135us) write_op.cc:276] Timestamp: P: 1714995597554573 usec, L: 0
0506 11:39:59.348493 (+ 479us) write_op.cc:270] Start()
0506 11:39:59.757653 (+409160us) write_op.cc:276] Timestamp: P: 1714995598195505 usec, L: 0
0506 11:39:59.757956 (+ 303us) log.cc:842] Serialized 708 byte log entry
0506 11:39:59.758303 (+ 347us) raft_consensus.cc:1631] Marking committed up to 19
0506 11:39:59.758637 (+ 334us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:39:59.758667 (+ 30us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:39:59.759080 (+ 413us) raft_consensus.cc:1671] finished
0506 11:39:59.759082 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:39:59.759128 (+ 46us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:59.348403 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:59.348505 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:59.348509 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:39:59.348511 (+ 2us) tablet.cc:660] Decoding operations
0506 11:39:59.348577 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:59.348588 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:59.348591 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:59.348630 (+ 39us) tablet.cc:699] Row locks acquired
0506 11:39:59.348632 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:59.758337 (+409705us) op_driver.cc:444] REPLICATION: finished
0506 11:39:59.758750 (+ 413us) write_op.cc:301] APPLY: starting
0506 11:39:59.758764 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 11:39:59.758795 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0506 11:39:59.758796 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:39:59.758853 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:39:59.758875 (+ 22us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:39:59.758908 (+ 33us) write_op.cc:312] APPLY: finished
0506 11:39:59.759027 (+ 119us) log.cc:842] Serialized 43 byte log entry
0506 11:39:59.759079 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:59.759107 (+ 28us) write_op.cc:454] Released schema lock
0506 11:39:59.759115 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0506 11:39:59.759208 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:39:59.759212 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:39:59.757853 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:59.757956 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:59.757961 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:59.757962 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:59.758055 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:59.758066 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:59.758068 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:59.758112 (+ 44us) tablet.cc:699] Row locks acquired
0506 11:39:59.758114 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":275,"threads_started":1,"wal-append.queue_time_us":349,"child_traces":[["op",{"apply.queue_time_us":320,"apply.run_cpu_time_us":535,"apply.run_wall_time_us":541,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":274,"prepare.run_cpu_time_us":312,"prepare.run_wall_time_us":312,"replication_time_us":1170442,"thread_start_us":431,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":358,"prepare.run_wall_time_us":358}]]}
I20240506 11:40:00.325619 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:00.325829 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:00.326556 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:00.327064 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:00.327101 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:00.327716 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:00.328104 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:00.328752 14168 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=20, attempt_no=0}) took 1699 ms. Trace:
I20240506 11:40:00.328872 14168 rpcz_store.cc:276] 0506 11:39:58.628873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:58.628939 (+ 66us) service_pool.cc:225] Handling call
0506 11:40:00.328698 (+1699759us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:59.007879 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:59.008013 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:59.008018 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:59.008019 (+ 1us) tablet.cc:660] Decoding operations
0506 11:39:59.008109 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:59.008124 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:59.008129 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:59.008185 (+ 56us) tablet.cc:699] Row locks acquired
0506 11:39:59.008187 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:59.008239 (+ 52us) write_op.cc:270] Start()
0506 11:39:59.760382 (+752143us) write_op.cc:276] Timestamp: P: 1714995599008197 usec, L: 0
0506 11:39:59.760387 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:39:59.760695 (+ 308us) log.cc:842] Serialized 354 byte log entry
0506 11:40:00.327666 (+566971us) op_driver.cc:444] REPLICATION: finished
0506 11:40:00.328141 (+ 475us) write_op.cc:301] APPLY: starting
0506 11:40:00.328159 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:00.328209 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:00.328239 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:00.328335 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:00.328358 (+ 23us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:00.328362 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:40:00.328478 (+ 116us) log.cc:842] Serialized 43 byte log entry
0506 11:40:00.328545 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:00.328564 (+ 19us) write_op.cc:454] Released schema lock
0506 11:40:00.328570 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":414,"num_ops":1,"prepare.queue_time_us":378564,"prepare.run_cpu_time_us":846,"prepare.run_wall_time_us":752910,"raft.queue_time_us":44,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":262,"replication_time_us":567253,"spinlock_wait_cycles":268160,"thread_start_us":318,"threads_started":1}]]}
I20240506 11:40:00.650561 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:00.650764 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:00.651860 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:00.652278 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:00.652487 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:00.652956 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:00.999202 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:00.999405 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:01.000216 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:01.000613 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:01.000808 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:01.001487 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:01.001950 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:01.002571 14175 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=21, attempt_no=0}) took 1653 ms. Trace:
I20240506 11:40:01.002662 14175 rpcz_store.cc:276] 0506 11:39:59.349295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:39:59.349363 (+ 68us) service_pool.cc:225] Handling call
0506 11:40:01.002520 (+1653157us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:39:59.760827 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:39:59.760953 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:39:59.760958 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:39:59.760961 (+ 3us) tablet.cc:660] Decoding operations
0506 11:39:59.761051 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:39:59.761062 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:39:59.761068 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:39:59.761117 (+ 49us) tablet.cc:699] Row locks acquired
0506 11:39:59.761119 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:39:59.761163 (+ 44us) write_op.cc:270] Start()
0506 11:39:59.809307 (+ 48144us) write_op.cc:276] Timestamp: P: 1714995599761118 usec, L: 0
0506 11:39:59.809311 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:39:59.809563 (+ 252us) log.cc:842] Serialized 354 byte log entry
0506 11:40:01.000630 (+1191067us) op_driver.cc:444] REPLICATION: finished
0506 11:40:01.001370 (+ 740us) write_op.cc:301] APPLY: starting
0506 11:40:01.001390 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:01.001432 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:01.001462 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:01.001532 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:01.001581 (+ 49us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:01.001589 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:40:01.002182 (+ 593us) log.cc:842] Serialized 43 byte log entry
0506 11:40:01.002420 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:01.002432 (+ 12us) write_op.cc:454] Released schema lock
0506 11:40:01.002437 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":652,"num_ops":1,"prepare.queue_time_us":411022,"prepare.run_cpu_time_us":741,"prepare.run_wall_time_us":48810,"replication_time_us":1191294,"thread_start_us":1338,"threads_started":1}]]}
I20240506 11:40:02.174738 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:02.174896 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:02.175799 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:02.176306 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:02.176295 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:02.177013 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:02.177277 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:02.177819 14182 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=22, attempt_no=0}) took 1847 ms. Trace:
I20240506 11:40:02.178036 14182 rpcz_store.cc:276] 0506 11:40:00.330767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:00.330836 (+ 69us) service_pool.cc:225] Handling call
0506 11:40:02.177724 (+1846888us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:00.331507 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:00.331602 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:00.331606 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:00.331607 (+ 1us) tablet.cc:660] Decoding operations
0506 11:40:00.331695 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:00.331704 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:00.331706 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:00.331747 (+ 41us) tablet.cc:699] Row locks acquired
0506 11:40:00.331748 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:40:00.331778 (+ 30us) write_op.cc:270] Start()
0506 11:40:01.292942 (+961164us) write_op.cc:276] Timestamp: P: 1714995600331762 usec, L: 0
0506 11:40:01.292947 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:40:01.293165 (+ 218us) log.cc:842] Serialized 354 byte log entry
0506 11:40:02.176361 (+883196us) op_driver.cc:444] REPLICATION: finished
0506 11:40:02.177160 (+ 799us) write_op.cc:301] APPLY: starting
0506 11:40:02.177178 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:02.177231 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:02.177286 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:02.177364 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:02.177396 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:02.177404 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:40:02.177510 (+ 106us) log.cc:842] Serialized 43 byte log entry
0506 11:40:02.177588 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:02.177606 (+ 18us) write_op.cc:454] Released schema lock
0506 11:40:02.177613 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":716,"num_ops":1,"prepare.queue_time_us":244,"prepare.run_cpu_time_us":678,"prepare.run_wall_time_us":961737,"replication_time_us":883389,"thread_start_us":816,"threads_started":2}]]}
I20240506 11:40:02.222071 14130 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:02.222290 14130 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:02.223024 14130 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:02.223513 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:02.223785 14182 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=23, attempt_no=0}) took 1219 ms. Trace:
I20240506 11:40:02.223991 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:02.223897 14182 rpcz_store.cc:276] 0506 11:40:01.004435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:01.004489 (+ 54us) service_pool.cc:225] Handling call
0506 11:40:02.223731 (+1219242us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:01.293270 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:01.293365 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:01.293370 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:40:01.293371 (+ 1us) tablet.cc:660] Decoding operations
0506 11:40:01.293468 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:01.293483 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:01.293490 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:01.293551 (+ 61us) tablet.cc:699] Row locks acquired
0506 11:40:01.293555 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:40:01.293615 (+ 60us) write_op.cc:270] Start()
0506 11:40:01.540770 (+247155us) write_op.cc:276] Timestamp: P: 1714995601293551 usec, L: 0
0506 11:40:01.540775 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:40:01.540991 (+ 216us) log.cc:842] Serialized 354 byte log entry
0506 11:40:02.223033 (+682042us) op_driver.cc:444] REPLICATION: finished
0506 11:40:02.223204 (+ 171us) write_op.cc:301] APPLY: starting
0506 11:40:02.223222 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:02.223260 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:02.223288 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:02.223363 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:02.223398 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:02.223405 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:40:02.223526 (+ 121us) log.cc:842] Serialized 43 byte log entry
0506 11:40:02.223593 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:02.223611 (+ 18us) write_op.cc:454] Released schema lock
0506 11:40:02.223617 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":113,"num_ops":1,"prepare.queue_time_us":288376,"prepare.run_cpu_time_us":707,"prepare.run_wall_time_us":247780,"replication_time_us":682226}]]}
I20240506 11:40:02.224571 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:02.224861 14130 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:02.588599 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:02.588810 13851 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.129:49185 (request call id 1372) took 1936 ms. Trace:
I20240506 11:40:02.588965 13851 rpcz_store.cc:276] 0506 11:40:00.652618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:00.652675 (+ 57us) service_pool.cc:225] Handling call
0506 11:40:00.652861 (+ 186us) raft_consensus.cc:1457] Updating replica for 2 ops
0506 11:40:00.653002 (+ 141us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 11:40:00.653005 (+ 3us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0506 11:40:00.653096 (+ 91us) write_op.cc:270] Start()
0506 11:40:01.597243 (+944147us) write_op.cc:276] Timestamp: P: 1714995599008197 usec, L: 0
0506 11:40:01.597853 (+ 610us) write_op.cc:270] Start()
0506 11:40:02.587009 (+989156us) write_op.cc:276] Timestamp: P: 1714995599761118 usec, L: 0
0506 11:40:02.587699 (+ 690us) log.cc:842] Serialized 708 byte log entry
0506 11:40:02.588160 (+ 461us) raft_consensus.cc:1631] Marking committed up to 22
0506 11:40:02.588480 (+ 320us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:40:02.588518 (+ 38us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:40:02.588681 (+ 163us) raft_consensus.cc:1671] finished
0506 11:40:02.588686 (+ 5us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:40:02.588737 (+ 51us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:01.597733 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:01.597843 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:01.597847 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:01.597849 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:01.597924 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:01.597938 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:01.597941 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:01.597986 (+ 45us) tablet.cc:699] Row locks acquired
0506 11:40:01.597987 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:40:02.588203 (+990216us) op_driver.cc:444] REPLICATION: finished
0506 11:40:02.588560 (+ 357us) write_op.cc:301] APPLY: starting
0506 11:40:02.588581 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:02.588623 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:02.588627 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:02.588718 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:02.588748 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:02.588806 (+ 58us) write_op.cc:312] APPLY: finished
0506 11:40:02.588906 (+ 100us) log.cc:842] Serialized 43 byte log entry
0506 11:40:02.588973 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:02.588989 (+ 16us) write_op.cc:454] Released schema lock
0506 11:40:02.588996 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0506 11:40:02.587525 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:02.587650 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:02.587658 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:40:02.587661 (+ 3us) tablet.cc:660] Decoding operations
0506 11:40:02.587761 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:02.587776 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:02.587781 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:02.587842 (+ 61us) tablet.cc:699] Row locks acquired
0506 11:40:02.587849 (+ 7us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":381,"threads_started":1,"wal-append.queue_time_us":451,"child_traces":[["op",{"apply.queue_time_us":287,"apply.run_cpu_time_us":631,"apply.run_wall_time_us":631,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":352,"prepare.run_cpu_time_us":365,"prepare.run_wall_time_us":366,"replication_time_us":1935087,"thread_start_us":434,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":335,"prepare.run_cpu_time_us":438,"prepare.run_wall_time_us":438,"thread_start_us":211,"threads_started":1}]]}
I20240506 11:40:03.394412 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:03.394618 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:03.395555 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:03.395939 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:03.395998 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:03.396559 13786 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:03.926515 14189 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:03.926743 14189 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:03.927601 14189 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:03.928067 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:03.928261 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:03.928987 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:03.929759 14196 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=24, attempt_no=0}) took 1749 ms. Trace:
I20240506 11:40:03.929924 14196 rpcz_store.cc:276] 0506 11:40:02.179766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:02.179857 (+ 91us) service_pool.cc:225] Handling call
0506 11:40:03.929689 (+1749832us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:02.180582 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:02.180680 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:02.180684 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:02.180685 (+ 1us) tablet.cc:660] Decoding operations
0506 11:40:02.180760 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:02.180769 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:02.180772 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:02.180808 (+ 36us) tablet.cc:699] Row locks acquired
0506 11:40:02.180809 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:40:02.180840 (+ 31us) write_op.cc:270] Start()
0506 11:40:02.949978 (+769138us) write_op.cc:276] Timestamp: P: 1714995602180800 usec, L: 0
0506 11:40:02.949982 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:40:02.950187 (+ 205us) log.cc:842] Serialized 354 byte log entry
0506 11:40:03.928386 (+978199us) op_driver.cc:444] REPLICATION: finished
0506 11:40:03.929024 (+ 638us) write_op.cc:301] APPLY: starting
0506 11:40:03.929043 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:03.929091 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:03.929131 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:03.929214 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:03.929245 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:03.929252 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:40:03.929376 (+ 124us) log.cc:842] Serialized 43 byte log entry
0506 11:40:03.929461 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:03.929486 (+ 25us) write_op.cc:454] Released schema lock
0506 11:40:03.929497 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":568,"num_ops":1,"prepare.queue_time_us":272,"prepare.run_cpu_time_us":909,"prepare.run_wall_time_us":769975,"raft.queue_time_us":228,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":978378,"thread_start_us":683,"threads_started":3}]]}
I20240506 11:40:03.936833 14189 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:03.938207 14196 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=25, attempt_no=0}) took 1712 ms. Trace:
I20240506 11:40:03.938302 14196 rpcz_store.cc:276] 0506 11:40:02.225879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:02.225937 (+ 58us) service_pool.cc:225] Handling call
0506 11:40:03.938151 (+1712214us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:02.950605 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:02.950747 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:02.950752 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:40:02.950754 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:02.950833 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:02.950843 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:02.950846 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:02.950926 (+ 80us) tablet.cc:699] Row locks acquired
0506 11:40:02.950928 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:02.951004 (+ 76us) write_op.cc:270] Start()
0506 11:40:03.303164 (+352160us) write_op.cc:276] Timestamp: P: 1714995602950969 usec, L: 0
0506 11:40:03.303169 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:40:03.303427 (+ 258us) log.cc:842] Serialized 354 byte log entry
0506 11:40:03.937624 (+634197us) op_driver.cc:444] REPLICATION: finished
0506 11:40:03.937732 (+ 108us) write_op.cc:301] APPLY: starting
0506 11:40:03.937748 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:03.937788 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:03.937817 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:03.937876 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:03.937900 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:03.937905 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:40:03.937969 (+ 64us) log.cc:842] Serialized 22 byte log entry
0506 11:40:03.938036 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:03.938051 (+ 15us) write_op.cc:454] Released schema lock
0506 11:40:03.938057 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"num_ops":1,"prepare.queue_time_us":724171,"prepare.run_cpu_time_us":839,"prepare.run_wall_time_us":352906,"replication_time_us":634437}]]}
I20240506 11:40:04.139690 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:04.549835 14189 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:04.550057 14189 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:04.550819 14189 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:04.551378 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:04.552026 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:04.553098 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:04.820835 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:04.821062 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:04.821810 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:04.822311 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:04.822476 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:04.823032 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:05.173931 14189 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:05.176291 14210 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=27, attempt_no=0}) took 1236 ms. Trace:
I20240506 11:40:05.176484 14210 rpcz_store.cc:276] 0506 11:40:03.940038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:03.940100 (+ 62us) service_pool.cc:225] Handling call
0506 11:40:05.176222 (+1236122us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:04.069254 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:04.069371 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:04.069378 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:40:04.069382 (+ 4us) tablet.cc:660] Decoding operations
0506 11:40:04.069469 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:04.069479 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:04.069483 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:04.069536 (+ 53us) tablet.cc:699] Row locks acquired
0506 11:40:04.069538 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:04.069584 (+ 46us) write_op.cc:270] Start()
0506 11:40:04.500724 (+431140us) write_op.cc:276] Timestamp: P: 1714995604069553 usec, L: 0
0506 11:40:04.500730 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:40:04.500974 (+ 244us) log.cc:842] Serialized 354 byte log entry
0506 11:40:05.175236 (+674262us) op_driver.cc:444] REPLICATION: finished
0506 11:40:05.175648 (+ 412us) write_op.cc:301] APPLY: starting
0506 11:40:05.175665 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:05.175704 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:05.175738 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:05.175829 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:05.175875 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:05.175882 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:40:05.175980 (+ 98us) log.cc:842] Serialized 22 byte log entry
0506 11:40:05.176099 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:05.176117 (+ 18us) write_op.cc:454] Released schema lock
0506 11:40:05.176124 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":353,"num_ops":1,"prepare.queue_time_us":128823,"prepare.run_cpu_time_us":736,"prepare.run_wall_time_us":431797,"replication_time_us":674481,"thread_start_us":279,"threads_started":1}]]}
I20240506 11:40:05.429951 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:06.031268 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:06.031471 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:06.032205 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:06.032672 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:06.032824 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:06.033437 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:06.033927 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:06.034456 14217 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=28, attempt_no=0}) took 1481 ms. Trace:
I20240506 11:40:06.034565 14217 rpcz_store.cc:276] 0506 11:40:04.553212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:04.553259 (+ 47us) service_pool.cc:225] Handling call
0506 11:40:06.034397 (+1481138us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:04.553714 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:04.553787 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:04.553791 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:04.553792 (+ 1us) tablet.cc:660] Decoding operations
0506 11:40:04.553849 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:04.553856 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:04.553858 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:04.553896 (+ 38us) tablet.cc:699] Row locks acquired
0506 11:40:04.553898 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:04.553925 (+ 27us) write_op.cc:270] Start()
0506 11:40:05.526044 (+972119us) write_op.cc:276] Timestamp: P: 1714995604553891 usec, L: 0
0506 11:40:05.526048 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:40:05.526254 (+ 206us) log.cc:842] Serialized 354 byte log entry
0506 11:40:06.033365 (+507111us) op_driver.cc:444] REPLICATION: finished
0506 11:40:06.033852 (+ 487us) write_op.cc:301] APPLY: starting
0506 11:40:06.033871 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:06.033913 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:06.033955 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:06.034051 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:06.034081 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:06.034087 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:40:06.034198 (+ 111us) log.cc:842] Serialized 43 byte log entry
0506 11:40:06.034268 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:06.034285 (+ 17us) write_op.cc:454] Released schema lock
0506 11:40:06.034291 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":400,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":972638,"raft.queue_time_us":35,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":187,"replication_time_us":507119,"thread_start_us":299,"threads_started":1}]]}
I20240506 11:40:06.255800 14189 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:06.255985 14189 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:06.256662 14189 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:06.257158 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:06.257187 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:06.257786 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:06.683554 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:06.683745 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:06.684453 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:06.684883 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:06.684885 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:06.685590 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:06.686173 14224 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=29, attempt_no=0}) took 1507 ms. Trace:
I20240506 11:40:06.686412 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:06.686340 14224 rpcz_store.cc:276] 0506 11:40:05.178334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:05.178406 (+ 72us) service_pool.cc:225] Handling call
0506 11:40:06.686117 (+1507711us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:05.526389 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:05.526496 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:05.526502 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:40:05.526504 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:05.526592 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:05.526603 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:05.526607 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:05.526658 (+ 51us) tablet.cc:699] Row locks acquired
0506 11:40:05.526660 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:05.526705 (+ 45us) write_op.cc:270] Start()
0506 11:40:06.131846 (+605141us) write_op.cc:276] Timestamp: P: 1714995605526673 usec, L: 0
0506 11:40:06.131853 (+ 7us) op_driver.cc:331] REPLICATION: starting
0506 11:40:06.132161 (+ 308us) log.cc:842] Serialized 354 byte log entry
0506 11:40:06.684955 (+552794us) op_driver.cc:444] REPLICATION: finished
0506 11:40:06.685458 (+ 503us) write_op.cc:301] APPLY: starting
0506 11:40:06.685475 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:06.685518 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:06.685547 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:06.685722 (+ 175us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:06.685758 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:06.685767 (+ 9us) write_op.cc:312] APPLY: finished
0506 11:40:06.685916 (+ 149us) log.cc:842] Serialized 22 byte log entry
0506 11:40:06.685995 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:06.686012 (+ 17us) write_op.cc:454] Released schema lock
0506 11:40:06.686019 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":422,"num_ops":1,"prepare.queue_time_us":347619,"prepare.run_cpu_time_us":799,"prepare.run_wall_time_us":605855,"replication_time_us":553078,"thread_start_us":322,"threads_started":1}]]}
I20240506 11:40:06.772316 14189 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:07.255025 14189 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:07.255203 14189 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:07.255950 14189 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:07.256498 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:07.256563 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:07.257431 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:07.257692 14189 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:07.258009 14228 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=30, attempt_no=0}) took 1221 ms. Trace:
I20240506 11:40:07.258123 14228 rpcz_store.cc:276] 0506 11:40:06.036265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:06.036333 (+ 68us) service_pool.cc:225] Handling call
0506 11:40:07.257949 (+1221616us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:06.132282 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:06.132399 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:06.132407 (+ 8us) write_op.cc:435] Acquired schema lock
0506 11:40:06.132410 (+ 3us) tablet.cc:660] Decoding operations
0506 11:40:06.132520 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:06.132539 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:06.132547 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:06.132614 (+ 67us) tablet.cc:699] Row locks acquired
0506 11:40:06.132617 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:40:06.132666 (+ 49us) write_op.cc:270] Start()
0506 11:40:06.785796 (+653130us) write_op.cc:276] Timestamp: P: 1714995606132625 usec, L: 0
0506 11:40:06.785800 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:40:06.785965 (+ 165us) log.cc:842] Serialized 354 byte log entry
0506 11:40:07.255991 (+470026us) op_driver.cc:444] REPLICATION: finished
0506 11:40:07.257132 (+ 1141us) write_op.cc:301] APPLY: starting
0506 11:40:07.257149 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:07.257188 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:07.257219 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:07.257325 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:07.257361 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:07.257372 (+ 11us) write_op.cc:312] APPLY: finished
0506 11:40:07.257463 (+ 91us) log.cc:842] Serialized 43 byte log entry
0506 11:40:07.257588 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:07.257818 (+ 230us) write_op.cc:454] Released schema lock
0506 11:40:07.257828 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1075,"num_ops":1,"prepare.queue_time_us":95541,"prepare.run_cpu_time_us":693,"prepare.run_wall_time_us":653760,"raft.queue_time_us":35,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":198,"replication_time_us":470164,"thread_start_us":979,"threads_started":1}]]}
I20240506 11:40:07.482833 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:07.483062 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:07.483850 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:07.484298 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:07.484403 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:07.484897 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:07.814883 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:07.816886 14231 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=31, attempt_no=0}) took 1128 ms. Trace:
I20240506 11:40:07.817025 14231 rpcz_store.cc:276] 0506 11:40:06.688233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:06.688306 (+ 73us) service_pool.cc:225] Handling call
0506 11:40:07.816803 (+1128497us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:06.786074 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:06.786185 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:06.786191 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:40:06.786193 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:06.786284 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:06.786295 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:06.786299 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:06.786351 (+ 52us) tablet.cc:699] Row locks acquired
0506 11:40:06.786353 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:06.786404 (+ 51us) write_op.cc:270] Start()
0506 11:40:07.406541 (+620137us) write_op.cc:276] Timestamp: P: 1714995606786359 usec, L: 0
0506 11:40:07.406545 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:40:07.406726 (+ 181us) log.cc:842] Serialized 354 byte log entry
0506 11:40:07.815849 (+409123us) op_driver.cc:444] REPLICATION: finished
0506 11:40:07.816264 (+ 415us) write_op.cc:301] APPLY: starting
0506 11:40:07.816281 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:07.816321 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:07.816351 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:07.816430 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:07.816458 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:07.816464 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:40:07.816595 (+ 131us) log.cc:842] Serialized 43 byte log entry
0506 11:40:07.816658 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:07.816675 (+ 17us) write_op.cc:454] Released schema lock
0506 11:40:07.816682 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":353,"num_ops":1,"prepare.queue_time_us":97388,"prepare.run_cpu_time_us":693,"prepare.run_wall_time_us":620749,"raft.queue_time_us":41,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":409278,"thread_start_us":468,"threads_started":1}]]}
I20240506 11:40:07.988612 14189 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:07.988816 14189 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:07.989668 14189 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:07.990170 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:07.990958 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:07.991580 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:08.691257 14189 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:08.693603 14237 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=32, attempt_no=0}) took 1433 ms. Trace:
I20240506 11:40:08.693713 14237 rpcz_store.cc:276] 0506 11:40:07.259783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:07.259857 (+ 74us) service_pool.cc:225] Handling call
0506 11:40:08.693538 (+1433681us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:07.406880 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:07.407009 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:07.407014 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:40:07.407015 (+ 1us) tablet.cc:660] Decoding operations
0506 11:40:07.407127 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:07.407144 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:07.407151 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:07.407204 (+ 53us) tablet.cc:699] Row locks acquired
0506 11:40:07.407206 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:07.407256 (+ 50us) write_op.cc:270] Start()
0506 11:40:07.740429 (+333173us) write_op.cc:276] Timestamp: P: 1714995607407193 usec, L: 0
0506 11:40:07.740434 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:40:07.740661 (+ 227us) log.cc:842] Serialized 354 byte log entry
0506 11:40:08.692591 (+951930us) op_driver.cc:444] REPLICATION: finished
0506 11:40:08.693012 (+ 421us) write_op.cc:301] APPLY: starting
0506 11:40:08.693029 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:08.693070 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:08.693119 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:08.693237 (+ 118us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:08.693269 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:08.693276 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:40:08.693382 (+ 106us) log.cc:842] Serialized 43 byte log entry
0506 11:40:08.693420 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:08.693434 (+ 14us) write_op.cc:454] Released schema lock
0506 11:40:08.693439 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":347,"num_ops":1,"prepare.queue_time_us":146566,"prepare.run_cpu_time_us":788,"prepare.run_wall_time_us":333851,"replication_time_us":952120,"thread_start_us":263,"threads_started":1}]]}
I20240506 11:40:08.740813 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:08.741037 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:08.741829 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:08.742273 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:08.742305 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:08.742900 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:09.135684 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:09.137200 14237 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=33, attempt_no=0}) took 1318 ms. Trace:
I20240506 11:40:09.137300 14237 rpcz_store.cc:276] 0506 11:40:07.818683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:07.818803 (+ 120us) service_pool.cc:225] Handling call
0506 11:40:09.137143 (+1318340us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:07.819274 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:07.819368 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:07.819372 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:07.819373 (+ 1us) tablet.cc:660] Decoding operations
0506 11:40:07.819439 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:07.819450 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:07.819456 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:07.819501 (+ 45us) tablet.cc:699] Row locks acquired
0506 11:40:07.819503 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:07.819536 (+ 33us) write_op.cc:270] Start()
0506 11:40:08.692669 (+873133us) write_op.cc:276] Timestamp: P: 1714995607819507 usec, L: 0
0506 11:40:08.692674 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:40:08.693138 (+ 464us) log.cc:842] Serialized 354 byte log entry
0506 11:40:09.136541 (+443403us) op_driver.cc:444] REPLICATION: finished
0506 11:40:09.136632 (+ 91us) write_op.cc:301] APPLY: starting
0506 11:40:09.136650 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:09.136695 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:09.136732 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:09.136820 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:09.136853 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:09.136862 (+ 9us) write_op.cc:312] APPLY: finished
0506 11:40:09.136955 (+ 93us) log.cc:842] Serialized 43 byte log entry
0506 11:40:09.137013 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:09.137030 (+ 17us) write_op.cc:454] Released schema lock
0506 11:40:09.137036 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"num_ops":1,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":801,"prepare.run_wall_time_us":873959,"replication_time_us":443843,"spinlock_wait_cycles":627072}]]}
I20240506 11:40:09.368286 14189 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:09.368476 14189 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:09.369357 14189 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:09.370128 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:09.370441 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:09.370888 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:09.371136 14189 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:09.882731 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:09.882930 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:09.883646 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:09.884066 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:09.884127 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:09.884898 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:09.885324 14245 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=34, attempt_no=0}) took 1189 ms. Trace:
I20240506 11:40:09.885433 14245 rpcz_store.cc:276] 0506 11:40:08.695751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:08.695809 (+ 58us) service_pool.cc:225] Handling call
0506 11:40:09.885278 (+1189469us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:08.696209 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:08.696287 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:08.696290 (+ 3us) write_op.cc:435] Acquired schema lock
0506 11:40:08.696292 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:08.696367 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:08.696375 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:08.696377 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:08.696410 (+ 33us) tablet.cc:699] Row locks acquired
0506 11:40:08.696411 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:40:08.696440 (+ 29us) write_op.cc:270] Start()
0506 11:40:09.479564 (+783124us) write_op.cc:276] Timestamp: P: 1714995608696411 usec, L: 0
0506 11:40:09.479569 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:40:09.479792 (+ 223us) log.cc:842] Serialized 354 byte log entry
0506 11:40:09.883940 (+404148us) op_driver.cc:444] REPLICATION: finished
0506 11:40:09.884716 (+ 776us) write_op.cc:301] APPLY: starting
0506 11:40:09.884736 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:09.884780 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:09.884808 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:09.884922 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:09.884957 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:09.884965 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:40:09.885069 (+ 104us) log.cc:842] Serialized 22 byte log entry
0506 11:40:09.885160 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:09.885177 (+ 17us) write_op.cc:454] Released schema lock
0506 11:40:09.885183 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":703,"num_ops":1,"prepare.queue_time_us":35,"prepare.run_cpu_time_us":3862,"prepare.run_wall_time_us":786925,"raft.queue_time_us":93,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":188,"replication_time_us":404346,"spinlock_wait_cycles":13696,"thread_start_us":3829,"threads_started":2}]]}
I20240506 11:40:10.223253 14189 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:10.223430 14189 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:10.224107 14189 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:10.224498 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:10.224551 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:10.225621 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:10.598428 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:10.600661 14251 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=35, attempt_no=0}) took 1461 ms. Trace:
I20240506 11:40:10.600816 14251 rpcz_store.cc:276] 0506 11:40:09.138997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:09.139058 (+ 61us) service_pool.cc:225] Handling call
0506 11:40:10.600600 (+1461542us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:09.483164 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:09.483263 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:09.483268 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:40:09.483269 (+ 1us) tablet.cc:660] Decoding operations
0506 11:40:09.483338 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:09.483348 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:09.483350 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:09.483391 (+ 41us) tablet.cc:699] Row locks acquired
0506 11:40:09.483392 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:40:09.483430 (+ 38us) write_op.cc:270] Start()
0506 11:40:09.764565 (+281135us) write_op.cc:276] Timestamp: P: 1714995609483388 usec, L: 0
0506 11:40:09.764570 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:40:09.764802 (+ 232us) log.cc:842] Serialized 354 byte log entry
0506 11:40:10.599639 (+834837us) op_driver.cc:444] REPLICATION: finished
0506 11:40:10.600022 (+ 383us) write_op.cc:301] APPLY: starting
0506 11:40:10.600039 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:10.600072 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:10.600118 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:10.600202 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:10.600233 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:10.600239 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:40:10.600361 (+ 122us) log.cc:842] Serialized 43 byte log entry
0506 11:40:10.600432 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:10.600449 (+ 17us) write_op.cc:454] Released schema lock
0506 11:40:10.600456 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":319,"num_ops":1,"prepare.queue_time_us":343690,"prepare.run_cpu_time_us":667,"prepare.run_wall_time_us":281727,"replication_time_us":835046,"thread_start_us":225,"threads_started":1}]]}
I20240506 11:40:10.669210 14189 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:11.561859 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:11.562057 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:11.562832 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:11.563338 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:11.563380 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:11.564080 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:11.564560 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:11.564582 14256 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=36, attempt_no=0}) took 1677 ms. Trace:
I20240506 11:40:11.564677 14256 rpcz_store.cc:276] 0506 11:40:09.886836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:09.886913 (+ 77us) service_pool.cc:225] Handling call
0506 11:40:11.564532 (+1677619us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:09.887381 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:09.887476 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:09.887480 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:09.887481 (+ 1us) tablet.cc:660] Decoding operations
0506 11:40:09.887540 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:09.887548 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:09.887550 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:09.887582 (+ 32us) tablet.cc:699] Row locks acquired
0506 11:40:09.887583 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:40:09.887614 (+ 31us) write_op.cc:270] Start()
0506 11:40:10.409778 (+522164us) write_op.cc:276] Timestamp: P: 1714995609887575 usec, L: 0
0506 11:40:10.409784 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:40:10.410024 (+ 240us) log.cc:842] Serialized 354 byte log entry
0506 11:40:11.563461 (+1153437us) op_driver.cc:444] REPLICATION: finished
0506 11:40:11.563960 (+ 499us) write_op.cc:301] APPLY: starting
0506 11:40:11.563976 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:11.564019 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:11.564043 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:11.564150 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:11.564185 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:11.564192 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:40:11.564304 (+ 112us) log.cc:842] Serialized 43 byte log entry
0506 11:40:11.564369 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:11.564386 (+ 17us) write_op.cc:454] Released schema lock
0506 11:40:11.564392 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":430,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":644,"prepare.run_wall_time_us":522704,"replication_time_us":1153651,"thread_start_us":308,"threads_started":1}]]}
I20240506 11:40:11.601203 14117 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:11.601405 14117 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:11.602103 14117 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:11.602599 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:11.602691 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:11.603503 13786 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:11.603899 14117 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:11.663115 14189 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:11.663347 14189 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:11.664065 14189 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:11.664501 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:11.665280 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:11.665831 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:12.446105 14189 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:12.486750 14261 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:12.487005 14261 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:12.487864 14261 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:12.488451 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:12.488629 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:12.489346 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:12.489609 14261 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:12.506454 14189 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:12.506656 14189 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:12.507416 14189 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:12.507921 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:12.507966 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:12.508975 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:12.509222 14189 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:12.716562 14189 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:12.716763 14189 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:12.717491 14189 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:12.717931 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:12.718127 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:12.718539 14265 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=39, attempt_no=0}) took 1114 ms. Trace:
I20240506 11:40:12.718971 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:12.718670 14265 rpcz_store.cc:276] 0506 11:40:11.604492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:11.604572 (+ 80us) service_pool.cc:225] Handling call
0506 11:40:12.718470 (+1113898us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:12.437815 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:12.437946 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:12.437952 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:40:12.437954 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:12.438057 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:12.438071 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:12.438078 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:12.438139 (+ 61us) tablet.cc:699] Row locks acquired
0506 11:40:12.438142 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:40:12.438195 (+ 53us) write_op.cc:270] Start()
0506 11:40:12.651354 (+213159us) write_op.cc:276] Timestamp: P: 1714995612438110 usec, L: 0
0506 11:40:12.651360 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:40:12.651567 (+ 207us) log.cc:842] Serialized 354 byte log entry
0506 11:40:12.717702 (+ 66135us) op_driver.cc:444] REPLICATION: finished
0506 11:40:12.717834 (+ 132us) write_op.cc:301] APPLY: starting
0506 11:40:12.717854 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:12.717896 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:12.717920 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:12.718046 (+ 126us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:12.718094 (+ 48us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:12.718104 (+ 10us) write_op.cc:312] APPLY: finished
0506 11:40:12.718226 (+ 122us) log.cc:842] Serialized 43 byte log entry
0506 11:40:12.718299 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:12.718319 (+ 20us) write_op.cc:454] Released schema lock
0506 11:40:12.718327 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38,"mutex_wait_us":25,"num_ops":1,"prepare.queue_time_us":832845,"prepare.run_cpu_time_us":805,"prepare.run_wall_time_us":213859,"raft.queue_time_us":231,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":310,"replication_time_us":66312,"spinlock_wait_cycles":56704}]]}
I20240506 11:40:12.719282 14189 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:12.886219 14261 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:12.886444 14261 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:12.887233 14261 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:12.887619 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:12.888254 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:12.888985 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:12.889302 14261 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:14.195835 14268 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:14.196031 14268 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:14.196802 14268 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:14.197253 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:14.197506 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:14.198017 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:14.198386 14268 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:14.199097 14273 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=40, attempt_no=0}) took 1707 ms. Trace:
I20240506 11:40:14.199273 14273 rpcz_store.cc:276] 0506 11:40:12.491414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:12.491491 (+ 77us) service_pool.cc:225] Handling call
0506 11:40:14.199048 (+1707557us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:12.651724 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:12.651863 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:12.651870 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:40:12.651872 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:12.651967 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:12.651978 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:12.651982 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:12.652029 (+ 47us) tablet.cc:699] Row locks acquired
0506 11:40:12.652042 (+ 13us) write_op.cc:260] PREPARE: finished
0506 11:40:12.652085 (+ 43us) write_op.cc:270] Start()
0506 11:40:13.571238 (+919153us) write_op.cc:276] Timestamp: P: 1714995612652053 usec, L: 0
0506 11:40:13.571244 (+ 6us) op_driver.cc:331] REPLICATION: starting
0506 11:40:13.571475 (+ 231us) log.cc:842] Serialized 354 byte log entry
0506 11:40:14.197806 (+626331us) op_driver.cc:444] REPLICATION: finished
0506 11:40:14.198418 (+ 612us) write_op.cc:301] APPLY: starting
0506 11:40:14.198437 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:14.198477 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:14.198501 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:14.198597 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:14.198630 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:14.198637 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:40:14.198740 (+ 103us) log.cc:842] Serialized 43 byte log entry
0506 11:40:14.198821 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:14.198840 (+ 19us) write_op.cc:454] Released schema lock
0506 11:40:14.198848 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":435,"num_ops":1,"prepare.queue_time_us":159776,"prepare.run_cpu_time_us":783,"prepare.run_wall_time_us":919851,"raft.queue_time_us":25,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":253,"replication_time_us":626502,"spinlock_wait_cycles":19712,"thread_start_us":332,"threads_started":1}]]}
I20240506 11:40:14.354382 14267 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:14.354588 14267 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:14.355468 14267 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:14.355902 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:14.356957 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:14.357470 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:14.756106 14268 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:14.756367 14268 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:14.757068 14268 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:14.757611 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:14.757614 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:14.758410 13786 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:14.758881 14280 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=41, attempt_no=0}) took 2038 ms. Trace:
I20240506 11:40:14.759047 14268 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:14.759017 14280 rpcz_store.cc:276] 0506 11:40:12.720768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:12.720835 (+ 67us) service_pool.cc:225] Handling call
0506 11:40:14.758817 (+2037982us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:13.571622 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:13.571738 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:13.571745 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:40:13.571747 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:13.571858 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:13.571871 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:13.571876 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:13.571930 (+ 54us) tablet.cc:699] Row locks acquired
0506 11:40:13.571932 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:13.571974 (+ 42us) write_op.cc:270] Start()
0506 11:40:13.892110 (+320136us) write_op.cc:276] Timestamp: P: 1714995613571953 usec, L: 0
0506 11:40:13.892115 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:40:13.892374 (+ 259us) log.cc:842] Serialized 354 byte log entry
0506 11:40:14.757282 (+864908us) op_driver.cc:444] REPLICATION: finished
0506 11:40:14.758247 (+ 965us) write_op.cc:301] APPLY: starting
0506 11:40:14.758266 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:14.758314 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:14.758338 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:14.758419 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:14.758455 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:14.758462 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:40:14.758546 (+ 84us) log.cc:842] Serialized 22 byte log entry
0506 11:40:14.758660 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:14.758677 (+ 17us) write_op.cc:454] Released schema lock
0506 11:40:14.758684 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":906,"num_ops":1,"prepare.queue_time_us":850390,"prepare.run_cpu_time_us":758,"prepare.run_wall_time_us":320820,"replication_time_us":865135,"thread_start_us":821,"threads_started":1}]]}
I20240506 11:40:14.876668 14267 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:15.501583 14267 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:15.501785 14267 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:15.502622 14267 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:15.503461 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:15.503844 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:15.504536 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:15.504693 14287 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=42, attempt_no=0}) took 1303 ms. Trace:
I20240506 11:40:15.504804 14287 rpcz_store.cc:276] 0506 11:40:14.200902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:14.200965 (+ 63us) service_pool.cc:225] Handling call
0506 11:40:15.504642 (+1303677us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:14.201395 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:14.201484 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:14.201487 (+ 3us) write_op.cc:435] Acquired schema lock
0506 11:40:14.201488 (+ 1us) tablet.cc:660] Decoding operations
0506 11:40:14.201549 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:14.201557 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:14.201559 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:14.201600 (+ 41us) tablet.cc:699] Row locks acquired
0506 11:40:14.201602 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:14.201638 (+ 36us) write_op.cc:270] Start()
0506 11:40:14.837779 (+636141us) write_op.cc:276] Timestamp: P: 1714995614201611 usec, L: 0
0506 11:40:14.837783 (+ 4us) op_driver.cc:331] REPLICATION: starting
0506 11:40:14.837947 (+ 164us) log.cc:842] Serialized 354 byte log entry
0506 11:40:15.503609 (+665662us) op_driver.cc:444] REPLICATION: finished
0506 11:40:15.504075 (+ 466us) write_op.cc:301] APPLY: starting
0506 11:40:15.504095 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:15.504137 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:15.504185 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:15.504271 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:15.504307 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:15.504314 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:40:15.504413 (+ 99us) log.cc:842] Serialized 43 byte log entry
0506 11:40:15.504482 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:15.504502 (+ 20us) write_op.cc:454] Released schema lock
0506 11:40:15.504508 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":380,"num_ops":1,"prepare.queue_time_us":36,"prepare.run_cpu_time_us":572,"prepare.run_wall_time_us":636649,"raft.queue_time_us":38,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":158,"replication_time_us":665782,"thread_start_us":303,"threads_started":1}]]}
I20240506 11:40:15.722767 14268 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:15.722965 14268 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:15.723627 14268 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:15.724020 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:15.724110 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:15.724664 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:15.921887 14268 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:15.923923 14287 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:50740 (ReqId={client: dae9131f6ee94285a2c5fcc4699a88fa, seq_no=43, attempt_no=0}) took 1163 ms. Trace:
I20240506 11:40:15.924062 14287 rpcz_store.cc:276] 0506 11:40:14.760744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:14.760811 (+ 67us) service_pool.cc:225] Handling call
0506 11:40:15.923848 (+1163037us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:14.838081 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
0506 11:40:14.838200 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:14.838207 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:40:14.838209 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:14.838299 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:14.838312 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:14.838318 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:14.838369 (+ 51us) tablet.cc:699] Row locks acquired
0506 11:40:14.838370 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:40:14.838408 (+ 38us) write_op.cc:270] Start()
0506 11:40:15.090551 (+252143us) write_op.cc:276] Timestamp: P: 1714995614838382 usec, L: 0
0506 11:40:15.090556 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:40:15.090773 (+ 217us) log.cc:842] Serialized 354 byte log entry
0506 11:40:15.923109 (+832336us) op_driver.cc:444] REPLICATION: finished
0506 11:40:15.923245 (+ 136us) write_op.cc:301] APPLY: starting
0506 11:40:15.923268 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:15.923317 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:15.923347 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:15.923438 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:15.923477 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:15.923485 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:40:15.923604 (+ 119us) log.cc:842] Serialized 43 byte log entry
0506 11:40:15.923676 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:15.923694 (+ 18us) write_op.cc:454] Released schema lock
0506 11:40:15.923721 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":64,"num_ops":1,"prepare.queue_time_us":76845,"prepare.run_cpu_time_us":712,"prepare.run_wall_time_us":252773,"replication_time_us":832525}]]}
I20240506 11:40:16.463958 14267 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:16.485653 14295 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:16.485913 14295 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:16.487041 14295 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:16.487052 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:16.487612 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:16.488137 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:16.488649 14295 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:16.648628 14267 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:16.648826 14267 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:16.649403 14267 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:16.649893 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:16.649978 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:16.650784 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:16.898797 14267 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:17.141723 14295 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:17.141930 14295 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:17.142649 14295 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:17.143126 13980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:17.143216 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:17.143771 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:17.334510 14267 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:17.334717 14267 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:17.335772 14267 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:17.336228 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:17.336320 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:17.337200 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:17.337435 14267 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:17.442205 13701 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240506 11:40:17.445261 13961 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:40:17.809273 14295 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:17.812250 13831 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
59610e18980643659c687eedd054f8f0 | 127.10.45.190:44121 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:44957 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
37ed28a1008e4e4c930eb4a15c156b9b | 127.10.45.129:33899 | HEALTHY | <none> | 1 | 0
d0a4d531336c4b4db79a960a8a241965 | 127.10.45.130:46563 | HEALTHY | <none> | 0 | 0
e1722c8227e641a5a4ac218853a639ab | 127.10.45.131:43815 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.45.129 | experimental | 127.10.45.129:33899
local_ip_for_outbound_sockets | 127.10.45.130 | experimental | 127.10.45.130:46563
local_ip_for_outbound_sockets | 127.10.45.131 | experimental | 127.10.45.131:43815
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.45.129:33899
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.45.130:46563
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.45.131:43815
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:44957 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240506 11:40:17.850628 10422 log_verifier.cc:126] Checking tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
I20240506 11:40:17.902173 10422 log_verifier.cc:177] Verified matching terms for 48 ops in tablet 7d05d3dc7cfc4d31aaaa2fded12e288d
I20240506 11:40:17.937682 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 13639
I20240506 11:40:17.949460 14337 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:17.951061 14337 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:17.952525 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:17.954330 14337 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:17.954538 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:17.955080 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:17.955519 14337 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:17.958704 13761 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:17.964388 14295 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:17.964545 14295 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:17.964785 14337 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:17.964916 14337 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:17.965164 14295 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:17.965487 14337 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:17.965919 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:17.966126 13850 raft_consensus.cc:2454] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1722c8227e641a5a4ac218853a639ab in term 1.
I20240506 11:40:17.966114 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:17.966328 13981 raft_consensus.cc:2454] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a4d531336c4b4db79a960a8a241965 in term 1.
I20240506 11:40:17.966528 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965, e1722c8227e641a5a4ac218853a639ab; no voters:
I20240506 11:40:17.966771 13786 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965, e1722c8227e641a5a4ac218853a639ab; no voters:
I20240506 11:40:17.966792 14337 raft_consensus.cc:2781] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 11:40:17.967020 14295 raft_consensus.cc:2781] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 11:40:17.967125 14295 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:17.967224 14295 raft_consensus.cc:3037] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:17.970283 14295 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:17.970862 14295 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 election: Requested vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:17.971072 14337 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37ed28a1008e4e4c930eb4a15c156b9b)
I20240506 11:40:17.971261 14337 raft_consensus.cc:3037] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:17.971280 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab"
W20240506 11:40:17.972942 13915 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.45.129:33899 timed out after 0.007s (SENT) [suppressed 1 similar messages]
W20240506 11:40:17.974260 13785 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.45.129:33899 timed out after 0.008s (SENT) [suppressed 1 similar messages]
I20240506 11:40:17.974185 14337 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:17.974654 13981 raft_consensus.cc:2379] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d0a4d531336c4b4db79a960a8a241965 in current term 2: Already voted for candidate e1722c8227e641a5a4ac218853a639ab in this term.
I20240506 11:40:17.974766 14337 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 election: Requested vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:17.975531 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965"
I20240506 11:40:17.975760 13850 raft_consensus.cc:2379] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e1722c8227e641a5a4ac218853a639ab in current term 2: Already voted for candidate d0a4d531336c4b4db79a960a8a241965 in this term.
W20240506 11:40:17.981385 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Timed out: RequestConsensusVote RPC to 127.10.45.129:33899 timed out after 0.007s (SENT)
W20240506 11:40:17.984323 13785 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Timed out: RequestConsensusVote RPC to 127.10.45.129:33899 timed out after 0.008s (SENT)
W20240506 11:40:17.988781 13785 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Timed out: RequestConsensusVote RPC to 127.10.45.129:33899 timed out after 0.008s (SENT)
I20240506 11:40:17.988924 13785 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, e1722c8227e641a5a4ac218853a639ab
I20240506 11:40:17.989194 14295 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240506 11:40:17.996140 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Timed out: RequestConsensusVote RPC to 127.10.45.129:33899 timed out after 0.008s (SENT)
I20240506 11:40:17.996277 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:17.996551 14337 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 11:40:17.997841 14295 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:40:17.998010 14295 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:17.998652 14295 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:17.999203 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 3 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab" is_pre_election: true
I20240506 11:40:17.999436 13981 raft_consensus.cc:2454] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a4d531336c4b4db79a960a8a241965 in term 2.
I20240506 11:40:17.999877 13786 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965, e1722c8227e641a5a4ac218853a639ab; no voters:
I20240506 11:40:18.000165 14295 raft_consensus.cc:2781] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 11:40:18.000274 14295 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:40:18.000383 14295 raft_consensus.cc:3037] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 2 FOLLOWER]: Advancing to term 3
I20240506 11:40:18.003101 14295 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:18.003839 14295 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 3 election: Requested vote from peers e1722c8227e641a5a4ac218853a639ab (127.10.45.131:43815), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:18.004148 13981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 3 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "e1722c8227e641a5a4ac218853a639ab"
I20240506 11:40:18.004323 13981 raft_consensus.cc:3037] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 2 FOLLOWER]: Advancing to term 3
I20240506 11:40:18.006909 13981 raft_consensus.cc:2454] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a4d531336c4b4db79a960a8a241965 in term 3.
I20240506 11:40:18.007331 13786 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0a4d531336c4b4db79a960a8a241965, e1722c8227e641a5a4ac218853a639ab; no voters:
I20240506 11:40:18.007560 14295 raft_consensus.cc:2781] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 3 FOLLOWER]: Leader election won for term 3
I20240506 11:40:18.008002 14295 raft_consensus.cc:686] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 3 LEADER]: Becoming Leader. State: Replica: d0a4d531336c4b4db79a960a8a241965, State: Running, Role: LEADER
I20240506 11:40:18.008493 14295 consensus_queue.cc:237] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 1.48, Last appended by leader: 48, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:18.011801 13579 catalog_manager.cc:5484] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 reported cstate change: term changed from 1 to 3, leader changed from 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129) to d0a4d531336c4b4db79a960a8a241965 (127.10.45.130). New cstate: current_term: 3 leader_uuid: "d0a4d531336c4b4db79a960a8a241965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } health_report { overall_health: UNKNOWN } } }
I20240506 11:40:18.013809 13981 raft_consensus.cc:1261] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Refusing update from remote peer d0a4d531336c4b4db79a960a8a241965: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 3 index: 49. (index mismatch)
I20240506 11:40:18.016321 14341 consensus_queue.cc:1035] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
W20240506 11:40:18.021382 13785 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Timed out: RequestConsensusVote RPC to 127.10.45.129:33899 timed out after 0.009s (SENT)
I20240506 11:40:18.023661 14337 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a4d531336c4b4db79a960a8a241965)
I20240506 11:40:18.023821 14337 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:18.024492 14337 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
I20240506 11:40:18.025009 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
W20240506 11:40:18.026818 13785 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Timed out: RequestConsensusVote RPC to 127.10.45.129:33899 timed out after 0.010s (SENT)
W20240506 11:40:18.041918 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Timed out: RequestConsensusVote RPC to 127.10.45.129:33899 timed out after 0.008s (SENT)
I20240506 11:40:18.042060 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:18.042320 14337 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 11:40:18.129681 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:18.130190 13721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "d0a4d531336c4b4db79a960a8a241965" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" is_pre_election: true
I20240506 11:40:18.131520 13980 raft_consensus.cc:1226] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Rejecting Update request from peer 37ed28a1008e4e4c930eb4a15c156b9b for earlier term 1. Current term is 3. Ops: []
I20240506 11:40:18.131876 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 13639
I20240506 11:40:18.132019 13850 raft_consensus.cc:1226] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 [term 3 LEADER]: Rejecting Update request from peer 37ed28a1008e4e4c930eb4a15c156b9b for earlier term 1. Current term is 3. Ops: []
W20240506 11:40:18.156556 13787 connection.cc:535] server connection from 127.10.45.129:49185 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 11:40:18.156880 13915 connection.cc:535] server connection from 127.10.45.129:48429 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 11:40:18.157207 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 13769
W20240506 11:40:18.158234 13785 consensus_peers.cc:485] T 7d05d3dc7cfc4d31aaaa2fded12e288d P d0a4d531336c4b4db79a960a8a241965 -> Peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Couldn't send request to peer 37ed28a1008e4e4c930eb4a15c156b9b. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:40:18.163794 13891 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:18.170655 14337 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a4d531336c4b4db79a960a8a241965)
I20240506 11:40:18.170802 14337 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:18.171537 14337 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
W20240506 11:40:18.173223 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33899: connect: Connection refused (error 111)
W20240506 11:40:18.185103 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563): Timed out: RequestConsensusVote RPC to 127.10.45.130:46563 timed out after 0.006s (SENT)
I20240506 11:40:18.185233 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:18.185513 14337 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 11:40:18.194258 14337 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a4d531336c4b4db79a960a8a241965)
I20240506 11:40:18.194420 14337 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:18.194988 14337 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
W20240506 11:40:18.196200 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33899: connect: Connection refused (error 111)
W20240506 11:40:18.210103 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563): Timed out: RequestConsensusVote RPC to 127.10.45.130:46563 timed out after 0.007s (SENT)
I20240506 11:40:18.210189 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:18.210444 14337 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 11:40:18.218197 14337 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a4d531336c4b4db79a960a8a241965)
I20240506 11:40:18.218348 14337 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:18.218923 14337 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
W20240506 11:40:18.220077 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33899: connect: Connection refused (error 111)
W20240506 11:40:18.233983 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563): Timed out: RequestConsensusVote RPC to 127.10.45.130:46563 timed out after 0.007s (SENT)
I20240506 11:40:18.234090 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:18.234364 14337 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 11:40:18.252583 14337 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a4d531336c4b4db79a960a8a241965)
I20240506 11:40:18.252764 14337 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:18.253360 14337 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
W20240506 11:40:18.254498 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33899: connect: Connection refused (error 111)
I20240506 11:40:18.271297 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
W20240506 11:40:18.270443 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563): Timed out: RequestConsensusVote RPC to 127.10.45.130:46563 timed out after 0.008s (SENT)
I20240506 11:40:18.271605 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:18.272092 14337 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 11:40:18.272523 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:18.272603 13849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:18.273280 13849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:18.273216 13848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:18.273636 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:18.273711 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:18.274058 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 13769
I20240506 11:40:18.274060 13848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d05d3dc7cfc4d31aaaa2fded12e288d" candidate_uuid: "e1722c8227e641a5a4ac218853a639ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "d0a4d531336c4b4db79a960a8a241965" is_pre_election: true
I20240506 11:40:18.281849 14337 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a4d531336c4b4db79a960a8a241965)
I20240506 11:40:18.282028 14337 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:18.282661 14337 leader_election.cc:288] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563), 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899)
W20240506 11:40:18.283871 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 37ed28a1008e4e4c930eb4a15c156b9b (127.10.45.129:33899): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33899: connect: Connection refused (error 111)
W20240506 11:40:18.294647 13915 connection.cc:535] client connection to 127.10.45.130:46563 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 11:40:18.294831 13915 leader_election.cc:334] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a4d531336c4b4db79a960a8a241965 (127.10.45.130:46563): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 11:40:18.294910 13917 connection.cc:535] server connection from 127.10.45.130:34025 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 11:40:18.294983 13915 leader_election.cc:302] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1722c8227e641a5a4ac218853a639ab; no voters: 37ed28a1008e4e4c930eb4a15c156b9b, d0a4d531336c4b4db79a960a8a241965
I20240506 11:40:18.295053 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 13899
I20240506 11:40:18.295300 14337 raft_consensus.cc:2726] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240506 11:40:18.301080 14021 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:18.408900 14337 raft_consensus.cc:482] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d0a4d531336c4b4db79a960a8a241965)
I20240506 11:40:18.409132 14337 raft_consensus.cc:504] T 7d05d3dc7cfc4d31aaaa2fded12e288d P e1722c8227e641a5a4ac218853a639ab [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1722c8227e641a5a4ac218853a639ab" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43815 } } peers { permanent_uuid: "d0a4d531336c4b4db79a960a8a241965" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46563 } } peers { permanent_uuid: "37ed28a1008e4e4c930eb4a15c156b9b" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33899 } }
I20240506 11:40:18.409715 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 13899
I20240506 11:40:18.431810 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 13549
I20240506 11:40:18.432847 13610 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:18.539175 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 13549
2024-05-06T11:40:18Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys (33979 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElection
I20240506 11:40:18.582401 10422 test_util.cc:268] Using random seed: -2002579608
I20240506 11:40:18.584120 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:40:18.584179 10422 ts_itest-base.cc:113] --------------
I20240506 11:40:18.584224 10422 ts_itest-base.cc:114] 5 tablet servers
I20240506 11:40:18.584273 10422 ts_itest-base.cc:115] 5 replicas per TS
I20240506 11:40:18.584332 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:40:18Z Disabled control of system clock
I20240506 11:40:18.611445 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:39273
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:39273 with env {}
W20240506 11:40:18.919490 14364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:18.919790 14364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:18.919924 14364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:18.926853 14364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:40:18.926970 14364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:18.927042 14364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:40:18.927114 14364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:40:18.935434 14364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:39273
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:39273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:18.936657 14364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:18.938247 14364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:18.949131 14369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:18.950008 14370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:18.950807 14372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:18.951704 14364 server_base.cc:1034] running on GCE node
I20240506 11:40:18.952366 14364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:18.953584 14364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:18.954777 14364 hybrid_clock.cc:648] HybridClock initialized: now 1714995618954723 us; error 36 us; skew 500 ppm
I20240506 11:40:18.955062 14364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:18.957088 14364 webserver.cc:460] Webserver started at http://127.10.45.190:43731/ using document root <none> and password file <none>
I20240506 11:40:18.957757 14364 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:18.957855 14364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:18.958120 14364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:18.960525 14364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "13bfa32bcf7a4f9787c1278d0e7d61da"
format_stamp: "Formatted at 2024-05-06 11:40:18 on dist-test-slave-7l0g"
I20240506 11:40:18.961215 14364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "13bfa32bcf7a4f9787c1278d0e7d61da"
format_stamp: "Formatted at 2024-05-06 11:40:18 on dist-test-slave-7l0g"
I20240506 11:40:18.965867 14364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240506 11:40:18.968967 14378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:18.970203 14364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 11:40:18.970342 14364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "13bfa32bcf7a4f9787c1278d0e7d61da"
format_stamp: "Formatted at 2024-05-06 11:40:18 on dist-test-slave-7l0g"
I20240506 11:40:18.970489 14364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:19.021117 14364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:19.021927 14364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:19.022162 14364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:19.038722 14364 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:39273
I20240506 11:40:19.038769 14429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:39273 every 8 connection(s)
I20240506 11:40:19.040067 14364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:40:19.043663 14430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:19.047704 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 14364
I20240506 11:40:19.047873 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:40:19.052855 14430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da: Bootstrap starting.
I20240506 11:40:19.056042 14430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:19.057276 14430 log.cc:824] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:19.059862 14430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da: No bootstrap required, opened a new log
I20240506 11:40:19.065629 14430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bfa32bcf7a4f9787c1278d0e7d61da" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 39273 } }
I20240506 11:40:19.065928 14430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:19.066015 14430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13bfa32bcf7a4f9787c1278d0e7d61da, State: Initialized, Role: FOLLOWER
I20240506 11:40:19.066540 14430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bfa32bcf7a4f9787c1278d0e7d61da" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 39273 } }
I20240506 11:40:19.066720 14430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:40:19.066830 14430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:40:19.066982 14430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:19.068771 14430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bfa32bcf7a4f9787c1278d0e7d61da" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 39273 } }
I20240506 11:40:19.069185 14430 leader_election.cc:302] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13bfa32bcf7a4f9787c1278d0e7d61da; no voters:
I20240506 11:40:19.069679 14430 leader_election.cc:288] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:40:19.069831 14435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:19.070416 14435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [term 1 LEADER]: Becoming Leader. State: Replica: 13bfa32bcf7a4f9787c1278d0e7d61da, State: Running, Role: LEADER
I20240506 11:40:19.070870 14435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bfa32bcf7a4f9787c1278d0e7d61da" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 39273 } }
I20240506 11:40:19.071497 14430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:40:19.073606 14437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13bfa32bcf7a4f9787c1278d0e7d61da. Latest consensus state: current_term: 1 leader_uuid: "13bfa32bcf7a4f9787c1278d0e7d61da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bfa32bcf7a4f9787c1278d0e7d61da" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 39273 } } }
I20240506 11:40:19.073491 14436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13bfa32bcf7a4f9787c1278d0e7d61da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bfa32bcf7a4f9787c1278d0e7d61da" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 39273 } } }
I20240506 11:40:19.073879 14437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:19.073880 14436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:19.076012 14442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:40:19.080083 14442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:40:19.087457 14442 catalog_manager.cc:1347] Generated new cluster ID: 499a48c651424ababe4bd7d7e456099d
I20240506 11:40:19.087580 14442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:40:19.098342 14442 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:40:19.099839 14442 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:40:19.115278 14442 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 13bfa32bcf7a4f9787c1278d0e7d61da: Generated new TSK 0
I20240506 11:40:19.116194 14442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:40:19.124709 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:39273
--builtin_ntp_servers=127.10.45.148:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20240506 11:40:19.421173 14454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:19.421437 14454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:19.421581 14454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:19.428300 14454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:19.428478 14454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:40:19.436358 14454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:39273
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:19.437520 14454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:19.439110 14454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:19.450280 14459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:19.451134 14460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:19.452034 14462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:19.452786 14454 server_base.cc:1034] running on GCE node
I20240506 11:40:19.453294 14454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:19.454070 14454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:19.455273 14454 hybrid_clock.cc:648] HybridClock initialized: now 1714995619455214 us; error 38 us; skew 500 ppm
I20240506 11:40:19.455516 14454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:19.457502 14454 webserver.cc:460] Webserver started at http://127.10.45.129:35333/ using document root <none> and password file <none>
I20240506 11:40:19.458104 14454 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:19.458221 14454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:19.458475 14454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:19.460768 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d0a7bb6d01d54afdaa32a7aa261cb623"
format_stamp: "Formatted at 2024-05-06 11:40:19 on dist-test-slave-7l0g"
I20240506 11:40:19.461463 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d0a7bb6d01d54afdaa32a7aa261cb623"
format_stamp: "Formatted at 2024-05-06 11:40:19 on dist-test-slave-7l0g"
I20240506 11:40:19.465976 14454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240506 11:40:19.468897 14468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:19.470045 14454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 11:40:19.470185 14454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d0a7bb6d01d54afdaa32a7aa261cb623"
format_stamp: "Formatted at 2024-05-06 11:40:19 on dist-test-slave-7l0g"
I20240506 11:40:19.470338 14454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:19.483680 14454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:19.484370 14454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:19.484608 14454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:19.485373 14454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:19.486724 14454 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:19.486799 14454 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:19.486914 14454 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:19.486987 14454 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240506 11:40:19.515661 14454 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:33513
I20240506 11:40:19.515702 14580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:33513 every 8 connection(s)
I20240506 11:40:19.516788 14454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:40:19.523772 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 14454
I20240506 11:40:19.523953 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:40:19.527419 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:39273
--builtin_ntp_servers=127.10.45.148:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20240506 11:40:19.528290 14581 heartbeater.cc:347] Connected to a master server at 127.10.45.190:39273
I20240506 11:40:19.528575 14581 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:19.529471 14581 heartbeater.cc:511] Master 127.10.45.190:39273 requested a full tablet report, sending...
I20240506 11:40:19.531664 14395 ts_manager.cc:194] Registered new tserver with Master: d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513)
I20240506 11:40:19.533555 14395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:49907
W20240506 11:40:19.822360 14584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:19.822600 14584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:19.822710 14584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:19.829708 14584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:19.829896 14584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:40:19.837958 14584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:39273
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:19.839087 14584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:19.840667 14584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:19.852079 14590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:19.852936 14592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:19.853668 14589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:19.854447 14584 server_base.cc:1034] running on GCE node
I20240506 11:40:19.855055 14584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:19.855808 14584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:19.857017 14584 hybrid_clock.cc:648] HybridClock initialized: now 1714995619856965 us; error 27 us; skew 500 ppm
I20240506 11:40:19.857271 14584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:19.859330 14584 webserver.cc:460] Webserver started at http://127.10.45.130:38619/ using document root <none> and password file <none>
I20240506 11:40:19.859944 14584 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:19.860055 14584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:19.860318 14584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:19.862776 14584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "680e178c01b14b439b3fe7e7d2a33038"
format_stamp: "Formatted at 2024-05-06 11:40:19 on dist-test-slave-7l0g"
I20240506 11:40:19.863526 14584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "680e178c01b14b439b3fe7e7d2a33038"
format_stamp: "Formatted at 2024-05-06 11:40:19 on dist-test-slave-7l0g"
I20240506 11:40:19.868235 14584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240506 11:40:19.871259 14598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:19.872526 14584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240506 11:40:19.872671 14584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "680e178c01b14b439b3fe7e7d2a33038"
format_stamp: "Formatted at 2024-05-06 11:40:19 on dist-test-slave-7l0g"
I20240506 11:40:19.872829 14584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:19.899272 14584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:19.900019 14584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:19.900288 14584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:19.901185 14584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:19.902645 14584 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:19.902724 14584 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:19.902855 14584 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:19.902909 14584 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:19.933871 14584 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:46181
I20240506 11:40:19.933915 14710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:46181 every 8 connection(s)
I20240506 11:40:19.935137 14584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:40:19.936038 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 14584
I20240506 11:40:19.936205 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:40:19.939625 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:39273
--builtin_ntp_servers=127.10.45.148:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20240506 11:40:19.948712 14711 heartbeater.cc:347] Connected to a master server at 127.10.45.190:39273
I20240506 11:40:19.948990 14711 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:19.949712 14711 heartbeater.cc:511] Master 127.10.45.190:39273 requested a full tablet report, sending...
I20240506 11:40:19.951006 14395 ts_manager.cc:194] Registered new tserver with Master: 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181)
I20240506 11:40:19.951714 14395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:50717
W20240506 11:40:20.246657 14714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:20.246927 14714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:20.247069 14714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:20.253780 14714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:20.253971 14714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:40:20.261693 14714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:39273
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:20.262822 14714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:20.264372 14714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:20.275730 14720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:20.276517 14722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:20.277266 14719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:20.278015 14714 server_base.cc:1034] running on GCE node
I20240506 11:40:20.278548 14714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:20.279392 14714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:20.280602 14714 hybrid_clock.cc:648] HybridClock initialized: now 1714995620280557 us; error 26 us; skew 500 ppm
I20240506 11:40:20.280875 14714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:20.282873 14714 webserver.cc:460] Webserver started at http://127.10.45.131:32947/ using document root <none> and password file <none>
I20240506 11:40:20.283540 14714 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:20.283664 14714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:20.283936 14714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:20.286222 14714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c1922e4786c34109bd610416f0f99872"
format_stamp: "Formatted at 2024-05-06 11:40:20 on dist-test-slave-7l0g"
I20240506 11:40:20.286906 14714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c1922e4786c34109bd610416f0f99872"
format_stamp: "Formatted at 2024-05-06 11:40:20 on dist-test-slave-7l0g"
I20240506 11:40:20.291467 14714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:40:20.294499 14728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:20.295727 14714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:20.295864 14714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c1922e4786c34109bd610416f0f99872"
format_stamp: "Formatted at 2024-05-06 11:40:20 on dist-test-slave-7l0g"
I20240506 11:40:20.296010 14714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:20.319167 14714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:20.319854 14714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:20.320093 14714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:20.320916 14714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:20.322288 14714 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:20.322364 14714 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:20.322486 14714 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:20.322539 14714 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:20.351851 14714 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:46109
I20240506 11:40:20.351884 14840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:46109 every 8 connection(s)
I20240506 11:40:20.353037 14714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:40:20.359258 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 14714
I20240506 11:40:20.359426 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:40:20.362308 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.132:0
--local_ip_for_outbound_sockets=127.10.45.132
--webserver_interface=127.10.45.132
--webserver_port=0
--tserver_master_addrs=127.10.45.190:39273
--builtin_ntp_servers=127.10.45.148:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20240506 11:40:20.364574 14841 heartbeater.cc:347] Connected to a master server at 127.10.45.190:39273
I20240506 11:40:20.364847 14841 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:20.365566 14841 heartbeater.cc:511] Master 127.10.45.190:39273 requested a full tablet report, sending...
I20240506 11:40:20.366748 14395 ts_manager.cc:194] Registered new tserver with Master: c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109)
I20240506 11:40:20.367502 14395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:43029
I20240506 11:40:20.536181 14581 heartbeater.cc:503] Master 127.10.45.190:39273 was elected leader, sending a full tablet report...
W20240506 11:40:20.662518 14844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:20.662772 14844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:20.662854 14844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:20.669548 14844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:20.669719 14844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.132
I20240506 11:40:20.677855 14844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.45.132
--webserver_port=0
--tserver_master_addrs=127.10.45.190:39273
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.132
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:20.678926 14844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:20.680486 14844 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:20.692070 14849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:20.692920 14850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:20.693717 14852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:20.694561 14844 server_base.cc:1034] running on GCE node
I20240506 11:40:20.695062 14844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:20.695802 14844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:20.697000 14844 hybrid_clock.cc:648] HybridClock initialized: now 1714995620696952 us; error 35 us; skew 500 ppm
I20240506 11:40:20.697230 14844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:20.699133 14844 webserver.cc:460] Webserver started at http://127.10.45.132:34865/ using document root <none> and password file <none>
I20240506 11:40:20.699739 14844 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:20.699862 14844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:20.700135 14844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:20.702359 14844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d78a1d347bc44a82b3cae400b801bf9d"
format_stamp: "Formatted at 2024-05-06 11:40:20 on dist-test-slave-7l0g"
I20240506 11:40:20.703069 14844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d78a1d347bc44a82b3cae400b801bf9d"
format_stamp: "Formatted at 2024-05-06 11:40:20 on dist-test-slave-7l0g"
I20240506 11:40:20.707377 14844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240506 11:40:20.710064 14858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:20.711194 14844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:20.711325 14844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d78a1d347bc44a82b3cae400b801bf9d"
format_stamp: "Formatted at 2024-05-06 11:40:20 on dist-test-slave-7l0g"
I20240506 11:40:20.711462 14844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:20.734908 14844 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:20.735613 14844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:20.735864 14844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:20.736613 14844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:20.737910 14844 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:20.737980 14844 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:20.738096 14844 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:20.738145 14844 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:20.765019 14844 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.132:45513
I20240506 11:40:20.765062 14970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.132:45513 every 8 connection(s)
I20240506 11:40:20.766076 14844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240506 11:40:20.769955 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 14844
I20240506 11:40:20.770097 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240506 11:40:20.772970 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.133:0
--local_ip_for_outbound_sockets=127.10.45.133
--webserver_interface=127.10.45.133
--webserver_port=0
--tserver_master_addrs=127.10.45.190:39273
--builtin_ntp_servers=127.10.45.148:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20240506 11:40:20.777801 14971 heartbeater.cc:347] Connected to a master server at 127.10.45.190:39273
I20240506 11:40:20.778034 14971 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:20.778832 14971 heartbeater.cc:511] Master 127.10.45.190:39273 requested a full tablet report, sending...
I20240506 11:40:20.780299 14395 ts_manager.cc:194] Registered new tserver with Master: d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513)
I20240506 11:40:20.780989 14395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.132:37339
I20240506 11:40:20.953914 14711 heartbeater.cc:503] Master 127.10.45.190:39273 was elected leader, sending a full tablet report...
W20240506 11:40:21.071836 14974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:21.072106 14974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:21.072233 14974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:21.078850 14974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:21.079037 14974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.133
I20240506 11:40:21.086850 14974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.10.45.133
--webserver_port=0
--tserver_master_addrs=127.10.45.190:39273
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.133
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:21.087982 14974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:21.089612 14974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:21.100557 14979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:21.101465 14982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:21.102229 14980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:21.103166 14974 server_base.cc:1034] running on GCE node
I20240506 11:40:21.103685 14974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:21.104465 14974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:21.105682 14974 hybrid_clock.cc:648] HybridClock initialized: now 1714995621105624 us; error 37 us; skew 500 ppm
I20240506 11:40:21.105993 14974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:21.107961 14974 webserver.cc:460] Webserver started at http://127.10.45.133:35241/ using document root <none> and password file <none>
I20240506 11:40:21.108606 14974 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:21.108721 14974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:21.109000 14974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:21.111416 14974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "84bc0a6756ff42b49941f4c11df1434f"
format_stamp: "Formatted at 2024-05-06 11:40:21 on dist-test-slave-7l0g"
I20240506 11:40:21.112145 14974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "84bc0a6756ff42b49941f4c11df1434f"
format_stamp: "Formatted at 2024-05-06 11:40:21 on dist-test-slave-7l0g"
I20240506 11:40:21.116778 14974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:40:21.119630 14988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:21.120817 14974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240506 11:40:21.120949 14974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "84bc0a6756ff42b49941f4c11df1434f"
format_stamp: "Formatted at 2024-05-06 11:40:21 on dist-test-slave-7l0g"
I20240506 11:40:21.121088 14974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:21.131585 14974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:21.132263 14974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:21.132493 14974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:21.133282 14974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:21.134529 14974 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:21.134606 14974 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:21.134729 14974 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:21.134774 14974 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:21.161005 14974 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.133:43953
I20240506 11:40:21.161041 15100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.133:43953 every 8 connection(s)
I20240506 11:40:21.162199 14974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240506 11:40:21.171906 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 14974
I20240506 11:40:21.172099 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240506 11:40:21.173036 15101 heartbeater.cc:347] Connected to a master server at 127.10.45.190:39273
I20240506 11:40:21.173301 15101 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:21.174302 15101 heartbeater.cc:511] Master 127.10.45.190:39273 requested a full tablet report, sending...
I20240506 11:40:21.175763 14395 ts_manager.cc:194] Registered new tserver with Master: 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
I20240506 11:40:21.176426 14395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.133:34671
I20240506 11:40:21.187268 10422 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240506 11:40:21.200891 14395 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45892:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20240506 11:40:21.205964 14395 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20240506 11:40:21.256166 14646 tablet_service.cc:1467] Processing CreateTablet for tablet f8cb8cbb86424c2d9ec72dc29b001ab0 (DEFAULT_TABLE table=TestTable [id=6eb9c61d3fb44b0392a4f23e757ea41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:21.255977 14516 tablet_service.cc:1467] Processing CreateTablet for tablet f8cb8cbb86424c2d9ec72dc29b001ab0 (DEFAULT_TABLE table=TestTable [id=6eb9c61d3fb44b0392a4f23e757ea41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:21.256304 14776 tablet_service.cc:1467] Processing CreateTablet for tablet f8cb8cbb86424c2d9ec72dc29b001ab0 (DEFAULT_TABLE table=TestTable [id=6eb9c61d3fb44b0392a4f23e757ea41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:21.257395 14646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cb8cbb86424c2d9ec72dc29b001ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:21.257409 14516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cb8cbb86424c2d9ec72dc29b001ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:21.257594 14776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cb8cbb86424c2d9ec72dc29b001ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:21.261507 15036 tablet_service.cc:1467] Processing CreateTablet for tablet f8cb8cbb86424c2d9ec72dc29b001ab0 (DEFAULT_TABLE table=TestTable [id=6eb9c61d3fb44b0392a4f23e757ea41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:21.262883 15036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cb8cbb86424c2d9ec72dc29b001ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:21.271142 14906 tablet_service.cc:1467] Processing CreateTablet for tablet f8cb8cbb86424c2d9ec72dc29b001ab0 (DEFAULT_TABLE table=TestTable [id=6eb9c61d3fb44b0392a4f23e757ea41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:21.272974 14906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cb8cbb86424c2d9ec72dc29b001ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:21.274988 15124 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Bootstrap starting.
I20240506 11:40:21.281816 15124 tablet_bootstrap.cc:654] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:21.282755 15123 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038: Bootstrap starting.
I20240506 11:40:21.282737 15125 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f: Bootstrap starting.
I20240506 11:40:21.286222 15123 tablet_bootstrap.cc:654] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:21.286777 15125 tablet_bootstrap.cc:654] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:21.287472 15126 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Bootstrap starting.
I20240506 11:40:21.287461 15124 log.cc:824] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:21.291035 15123 log.cc:824] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:21.291296 15125 log.cc:824] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:21.293179 15126 tablet_bootstrap.cc:654] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:21.294461 15126 log.cc:824] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:21.296866 15124 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: No bootstrap required, opened a new log
I20240506 11:40:21.297209 15124 ts_tablet_manager.cc:1367] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.011s
I20240506 11:40:21.298867 15126 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: No bootstrap required, opened a new log
I20240506 11:40:21.301761 15125 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f: No bootstrap required, opened a new log
I20240506 11:40:21.302099 15125 ts_tablet_manager.cc:1367] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20240506 11:40:21.305946 15124 raft_consensus.cc:348] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.307068 15124 raft_consensus.cc:374] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:21.307217 15126 ts_tablet_manager.cc:1367] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.008s
I20240506 11:40:21.307219 15124 raft_consensus.cc:729] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1922e4786c34109bd610416f0f99872, State: Initialized, Role: FOLLOWER
I20240506 11:40:21.307960 15124 consensus_queue.cc:260] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.308856 15123 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038: No bootstrap required, opened a new log
I20240506 11:40:21.309229 15123 ts_tablet_manager.cc:1367] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.013s
I20240506 11:40:21.312212 15127 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d: Bootstrap starting.
I20240506 11:40:21.316004 15124 ts_tablet_manager.cc:1398] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Time spent starting tablet: real 0.019s user 0.014s sys 0.002s
I20240506 11:40:21.316352 14841 heartbeater.cc:503] Master 127.10.45.190:39273 was elected leader, sending a full tablet report...
I20240506 11:40:21.317749 15127 tablet_bootstrap.cc:654] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:21.318099 15123 raft_consensus.cc:348] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.318536 15123 raft_consensus.cc:374] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:21.318660 15123 raft_consensus.cc:729] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680e178c01b14b439b3fe7e7d2a33038, State: Initialized, Role: FOLLOWER
I20240506 11:40:21.318416 15126 raft_consensus.cc:348] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.318795 15126 raft_consensus.cc:374] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:21.318810 15127 log.cc:824] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:21.318909 15126 raft_consensus.cc:729] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a7bb6d01d54afdaa32a7aa261cb623, State: Initialized, Role: FOLLOWER
I20240506 11:40:21.319456 15123 consensus_queue.cc:260] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.319558 15126 consensus_queue.cc:260] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.320269 15125 raft_consensus.cc:348] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.321058 15125 raft_consensus.cc:374] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:21.321298 15125 raft_consensus.cc:729] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bc0a6756ff42b49941f4c11df1434f, State: Initialized, Role: FOLLOWER
I20240506 11:40:21.322091 15125 consensus_queue.cc:260] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.323071 15101 heartbeater.cc:503] Master 127.10.45.190:39273 was elected leader, sending a full tablet report...
I20240506 11:40:21.323602 15125 ts_tablet_manager.cc:1398] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f: Time spent starting tablet: real 0.021s user 0.003s sys 0.007s
I20240506 11:40:21.326541 15127 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d: No bootstrap required, opened a new log
I20240506 11:40:21.326859 15127 ts_tablet_manager.cc:1367] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240506 11:40:21.332270 15126 ts_tablet_manager.cc:1398] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Time spent starting tablet: real 0.025s user 0.015s sys 0.006s
I20240506 11:40:21.332608 15123 ts_tablet_manager.cc:1398] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038: Time spent starting tablet: real 0.023s user 0.015s sys 0.007s
I20240506 11:40:21.334249 15127 raft_consensus.cc:348] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.334580 15127 raft_consensus.cc:374] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:21.334681 15127 raft_consensus.cc:729] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78a1d347bc44a82b3cae400b801bf9d, State: Initialized, Role: FOLLOWER
I20240506 11:40:21.335258 15127 consensus_queue.cc:260] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.336231 14971 heartbeater.cc:503] Master 127.10.45.190:39273 was elected leader, sending a full tablet report...
I20240506 11:40:21.336536 15127 ts_tablet_manager.cc:1398] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20240506 11:40:21.354934 14842 tablet.cc:2369] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:21.403396 15132 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:40:21.403618 15132 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.405390 15132 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
W20240506 11:40:21.413635 15102 tablet.cc:2369] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:21.415828 14666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "c1922e4786c34109bd610416f0f99872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038" is_pre_election: true
I20240506 11:40:21.416424 14666 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1922e4786c34109bd610416f0f99872 in term 0.
I20240506 11:40:21.416702 15056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "c1922e4786c34109bd610416f0f99872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f" is_pre_election: true
I20240506 11:40:21.417136 15056 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1922e4786c34109bd610416f0f99872 in term 0.
I20240506 11:40:21.417502 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "c1922e4786c34109bd610416f0f99872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d78a1d347bc44a82b3cae400b801bf9d" is_pre_election: true
I20240506 11:40:21.418027 14926 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1922e4786c34109bd610416f0f99872 in term 0.
I20240506 11:40:21.418475 14731 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038, 84bc0a6756ff42b49941f4c11df1434f, c1922e4786c34109bd610416f0f99872; no voters:
I20240506 11:40:21.418879 15132 raft_consensus.cc:2781] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 11:40:21.419020 15132 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:40:21.419131 15132 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:21.419570 14536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "c1922e4786c34109bd610416f0f99872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" is_pre_election: true
I20240506 11:40:21.420011 14536 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1922e4786c34109bd610416f0f99872 in term 0.
I20240506 11:40:21.422535 15132 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.423594 14666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "c1922e4786c34109bd610416f0f99872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038"
I20240506 11:40:21.423624 14536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "c1922e4786c34109bd610416f0f99872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623"
I20240506 11:40:21.423668 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "c1922e4786c34109bd610416f0f99872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d78a1d347bc44a82b3cae400b801bf9d"
I20240506 11:40:21.423700 15132 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [CANDIDATE]: Term 1 election: Requested vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
I20240506 11:40:21.423805 14536 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:21.423830 14926 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:21.423813 14666 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:21.425662 14536 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1922e4786c34109bd610416f0f99872 in term 1.
I20240506 11:40:21.426308 15056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "c1922e4786c34109bd610416f0f99872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f"
I20240506 11:40:21.426450 15056 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:21.426697 14666 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1922e4786c34109bd610416f0f99872 in term 1.
I20240506 11:40:21.426697 14926 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1922e4786c34109bd610416f0f99872 in term 1.
I20240506 11:40:21.427168 14729 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: c1922e4786c34109bd610416f0f99872, d0a7bb6d01d54afdaa32a7aa261cb623, d78a1d347bc44a82b3cae400b801bf9d; no voters:
I20240506 11:40:21.427485 15132 raft_consensus.cc:2781] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:21.427870 15132 raft_consensus.cc:686] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 1 LEADER]: Becoming Leader. State: Replica: c1922e4786c34109bd610416f0f99872, State: Running, Role: LEADER
I20240506 11:40:21.428364 15132 consensus_queue.cc:237] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:21.428750 15056 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1922e4786c34109bd610416f0f99872 in term 1.
I20240506 11:40:21.432827 14395 catalog_manager.cc:5484] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1922e4786c34109bd610416f0f99872 (127.10.45.131). New cstate: current_term: 1 leader_uuid: "c1922e4786c34109bd610416f0f99872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } health_report { overall_health: UNKNOWN } } }
W20240506 11:40:21.437532 14712 tablet.cc:2369] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:21.443703 10422 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240506 11:40:21.445487 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d0a7bb6d01d54afdaa32a7aa261cb623 to finish bootstrapping
I20240506 11:40:21.453205 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 680e178c01b14b439b3fe7e7d2a33038 to finish bootstrapping
I20240506 11:40:21.460465 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c1922e4786c34109bd610416f0f99872 to finish bootstrapping
I20240506 11:40:21.467233 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d78a1d347bc44a82b3cae400b801bf9d to finish bootstrapping
I20240506 11:40:21.473790 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 84bc0a6756ff42b49941f4c11df1434f to finish bootstrapping
I20240506 11:40:21.489928 10422 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (5 alive)...
I20240506 11:40:21.510094 14926 raft_consensus.cc:1261] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 1 FOLLOWER]: Refusing update from remote peer c1922e4786c34109bd610416f0f99872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:21.510094 15056 raft_consensus.cc:1261] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 1 FOLLOWER]: Refusing update from remote peer c1922e4786c34109bd610416f0f99872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:21.510524 14666 raft_consensus.cc:1261] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 1 FOLLOWER]: Refusing update from remote peer c1922e4786c34109bd610416f0f99872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:21.511255 15132 consensus_queue.cc:1035] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:21.511562 14536 raft_consensus.cc:1261] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 1 FOLLOWER]: Refusing update from remote peer c1922e4786c34109bd610416f0f99872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:21.511765 15149 consensus_queue.cc:1035] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:21.512184 15132 consensus_queue.cc:1035] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:21.512389 15142 consensus_queue.cc:1035] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 11:40:21.517829 14972 tablet.cc:2369] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 11:40:21.522269 14582 tablet.cc:2369] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:21.544803 15153 mvcc.cc:205] Tried to move back new op lower bound from 7024622065694429184 to 7024622065375686656. Current Snapshot: MvccSnapshot[applied={T|T < 7024622065694429184}]
I20240506 11:40:21.587563 10422 raft_consensus_election-itest.cc:408] Killing current leader c1922e4786c34109bd610416f0f99872...
I20240506 11:40:21.587747 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 14714
I20240506 11:40:21.588721 14836 generic_service.cc:196] Checking for leaks (request via RPC)
W20240506 11:40:21.675503 14730 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.10.45.130:46181 timed out after 0.050s (SENT)
I20240506 11:40:21.676616 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 14714
I20240506 11:40:21.694617 10422 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20240506 11:40:21.697031 10422 test_util.cc:268] Using random seed: -1999464975
I20240506 11:40:21.821777 10422 test_util.cc:268] Using random seed: -1999340257
I20240506 11:40:22.028712 10422 test_util.cc:268] Using random seed: -1999133300
I20240506 11:40:22.335907 10422 test_util.cc:268] Using random seed: -1998826124
I20240506 11:40:22.743204 10422 test_util.cc:268] Using random seed: -1998418805
I20240506 11:40:23.088213 15167 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1922e4786c34109bd610416f0f99872)
I20240506 11:40:23.088383 15168 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1922e4786c34109bd610416f0f99872)
I20240506 11:40:23.088430 15167 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:23.088644 15168 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:23.089494 15169 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1922e4786c34109bd610416f0f99872)
I20240506 11:40:23.089778 15169 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:23.095779 15167 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513)
I20240506 11:40:23.107285 15168 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
I20240506 11:40:23.112624 15176 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1922e4786c34109bd610416f0f99872)
I20240506 11:40:23.112902 15176 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
W20240506 11:40:23.114161 14472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:23.116730 15169 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
W20240506 11:40:23.118345 14472 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
W20240506 11:40:23.130867 14992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:23.132306 14535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "680e178c01b14b439b3fe7e7d2a33038" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" is_pre_election: true
I20240506 11:40:23.132580 14535 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680e178c01b14b439b3fe7e7d2a33038 in term 1.
I20240506 11:40:23.135143 14666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038" is_pre_election: true
I20240506 11:40:23.135371 14666 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a7bb6d01d54afdaa32a7aa261cb623 in term 1.
I20240506 11:40:23.140725 15176 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
W20240506 11:40:23.141669 14602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:23.144932 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d78a1d347bc44a82b3cae400b801bf9d" is_pre_election: true
I20240506 11:40:23.145228 14925 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a7bb6d01d54afdaa32a7aa261cb623 in term 1.
I20240506 11:40:23.145828 14469 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038, d0a7bb6d01d54afdaa32a7aa261cb623, d78a1d347bc44a82b3cae400b801bf9d; no voters: c1922e4786c34109bd610416f0f99872
I20240506 11:40:23.146390 15168 raft_consensus.cc:2781] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 11:40:23.146553 15168 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1922e4786c34109bd610416f0f99872)
I20240506 11:40:23.146672 15168 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:23.149148 14666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "84bc0a6756ff42b49941f4c11df1434f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038" is_pre_election: true
I20240506 11:40:23.149363 14666 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84bc0a6756ff42b49941f4c11df1434f in term 1.
I20240506 11:40:23.149657 15168 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:23.150684 15168 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [CANDIDATE]: Term 2 election: Requested vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
I20240506 11:40:23.151361 14666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038"
I20240506 11:40:23.151515 14666 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:23.151858 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d78a1d347bc44a82b3cae400b801bf9d"
I20240506 11:40:23.152025 14925 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:23.154189 14666 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a7bb6d01d54afdaa32a7aa261cb623 in term 2.
I20240506 11:40:23.154938 14925 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a7bb6d01d54afdaa32a7aa261cb623 in term 2.
I20240506 11:40:23.155328 14469 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038, d0a7bb6d01d54afdaa32a7aa261cb623, d78a1d347bc44a82b3cae400b801bf9d; no voters:
I20240506 11:40:23.155691 15168 raft_consensus.cc:2781] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 11:40:23.156635 14666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038" is_pre_election: true
I20240506 11:40:23.156961 14666 raft_consensus.cc:2379] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d78a1d347bc44a82b3cae400b801bf9d in current term 2: Already voted for candidate d0a7bb6d01d54afdaa32a7aa261cb623 in this term.
I20240506 11:40:23.160692 15056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f" is_pre_election: true
I20240506 11:40:23.160918 15056 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0a7bb6d01d54afdaa32a7aa261cb623 in term 1.
I20240506 11:40:23.161753 15056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f"
I20240506 11:40:23.161947 15056 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:23.162129 14535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "84bc0a6756ff42b49941f4c11df1434f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" is_pre_election: true
I20240506 11:40:23.163781 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "84bc0a6756ff42b49941f4c11df1434f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d78a1d347bc44a82b3cae400b801bf9d" is_pre_election: true
W20240506 11:40:23.164889 14602 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:23.164904 15055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "680e178c01b14b439b3fe7e7d2a33038" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f" is_pre_election: true
I20240506 11:40:23.165141 15056 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0a7bb6d01d54afdaa32a7aa261cb623 in term 2.
I20240506 11:40:23.165566 14925 raft_consensus.cc:2379] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84bc0a6756ff42b49941f4c11df1434f in current term 2: Already voted for candidate d0a7bb6d01d54afdaa32a7aa261cb623 in this term.
I20240506 11:40:23.165638 15168 raft_consensus.cc:686] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 2 LEADER]: Becoming Leader. State: Replica: d0a7bb6d01d54afdaa32a7aa261cb623, State: Running, Role: LEADER
W20240506 11:40:23.165946 14992 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:23.166200 15168 consensus_queue.cc:237] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:23.167620 14536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" is_pre_election: true
I20240506 11:40:23.167654 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "680e178c01b14b439b3fe7e7d2a33038" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d78a1d347bc44a82b3cae400b801bf9d" is_pre_election: true
I20240506 11:40:23.167860 14925 raft_consensus.cc:2379] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 680e178c01b14b439b3fe7e7d2a33038 in current term 2: Already voted for candidate d0a7bb6d01d54afdaa32a7aa261cb623 in this term.
W20240506 11:40:23.169278 14472 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:23.169457 14599 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038, d0a7bb6d01d54afdaa32a7aa261cb623; no voters: 84bc0a6756ff42b49941f4c11df1434f, c1922e4786c34109bd610416f0f99872, d78a1d347bc44a82b3cae400b801bf9d
W20240506 11:40:23.165566 14862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:23.170076 15055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f" is_pre_election: true
I20240506 11:40:23.170240 15176 raft_consensus.cc:2726] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:23.170331 15055 raft_consensus.cc:2379] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d78a1d347bc44a82b3cae400b801bf9d in current term 2: Already voted for candidate d0a7bb6d01d54afdaa32a7aa261cb623 in this term.
I20240506 11:40:23.171000 14861 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d78a1d347bc44a82b3cae400b801bf9d; no voters: 680e178c01b14b439b3fe7e7d2a33038, 84bc0a6756ff42b49941f4c11df1434f, d0a7bb6d01d54afdaa32a7aa261cb623
I20240506 11:40:23.171514 15169 raft_consensus.cc:2726] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240506 11:40:23.173436 14862 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:23.174607 14990 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038, 84bc0a6756ff42b49941f4c11df1434f; no voters: c1922e4786c34109bd610416f0f99872, d0a7bb6d01d54afdaa32a7aa261cb623, d78a1d347bc44a82b3cae400b801bf9d
I20240506 11:40:23.174981 15167 raft_consensus.cc:2726] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:23.175446 14395 catalog_manager.cc:5484] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 reported cstate change: term changed from 1 to 2, leader changed from c1922e4786c34109bd610416f0f99872 (127.10.45.131) to d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129). New cstate: current_term: 2 leader_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } health_report { overall_health: UNKNOWN } } }
I20240506 11:40:23.250823 10422 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (4 alive)...
W20240506 11:40:23.253952 15107 meta_cache.cc:302] tablet f8cb8cbb86424c2d9ec72dc29b001ab0: replica c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109) has failed: Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
W20240506 11:40:23.264140 14626 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45136: Illegal state: replica 680e178c01b14b439b3fe7e7d2a33038 is not leader of this config: current role FOLLOWER
I20240506 11:40:23.277835 14666 raft_consensus.cc:1261] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Refusing update from remote peer d0a7bb6d01d54afdaa32a7aa261cb623: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240506 11:40:23.277850 14925 raft_consensus.cc:1261] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Refusing update from remote peer d0a7bb6d01d54afdaa32a7aa261cb623: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240506 11:40:23.277947 15055 raft_consensus.cc:1261] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Refusing update from remote peer d0a7bb6d01d54afdaa32a7aa261cb623: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240506 11:40:23.278530 15198 consensus_queue.cc:1035] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240506 11:40:23.279104 15198 consensus_queue.cc:1035] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240506 11:40:23.279490 14472 consensus_peers.cc:485] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 -> Peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Couldn't send request to peer c1922e4786c34109bd610416f0f99872. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:40:23.279567 15208 consensus_queue.cc:1035] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240506 11:40:23.292340 15210 mvcc.cc:205] Tried to move back new op lower bound from 7024622072938209280 to 7024622072494833664. Current Snapshot: MvccSnapshot[applied={T|T < 7024622072938209280}]
I20240506 11:40:23.319415 10422 raft_consensus_election-itest.cc:408] Killing current leader d0a7bb6d01d54afdaa32a7aa261cb623...
I20240506 11:40:23.319588 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 14454
I20240506 11:40:23.320515 14576 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:23.407711 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 14454
I20240506 11:40:23.424659 10422 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20240506 11:40:23.427417 10422 test_util.cc:268] Using random seed: -1997734589
I20240506 11:40:23.534148 10422 test_util.cc:268] Using random seed: -1997627887
I20240506 11:40:23.740712 10422 test_util.cc:268] Using random seed: -1997421300
I20240506 11:40:24.047577 10422 test_util.cc:268] Using random seed: -1997114454
I20240506 11:40:24.454922 10422 test_util.cc:268] Using random seed: -1996707094
I20240506 11:40:24.820478 15224 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a7bb6d01d54afdaa32a7aa261cb623)
I20240506 11:40:24.820693 15224 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:24.821540 15224 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
I20240506 11:40:24.822228 15055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f" is_pre_election: true
I20240506 11:40:24.822510 15055 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78a1d347bc44a82b3cae400b801bf9d in term 2.
I20240506 11:40:24.827392 15225 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a7bb6d01d54afdaa32a7aa261cb623)
I20240506 11:40:24.827721 15226 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d0a7bb6d01d54afdaa32a7aa261cb623)
I20240506 11:40:24.827656 14665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038" is_pre_election: true
I20240506 11:40:24.827890 15226 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:24.828140 15225 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:24.828456 14665 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78a1d347bc44a82b3cae400b801bf9d in term 2.
I20240506 11:40:24.828998 14665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "84bc0a6756ff42b49941f4c11df1434f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038" is_pre_election: true
I20240506 11:40:24.829165 14665 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84bc0a6756ff42b49941f4c11df1434f in term 2.
I20240506 11:40:24.836333 15225 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
I20240506 11:40:24.836333 15055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "680e178c01b14b439b3fe7e7d2a33038" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f" is_pre_election: true
I20240506 11:40:24.836575 15055 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680e178c01b14b439b3fe7e7d2a33038 in term 2.
I20240506 11:40:24.846017 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "84bc0a6756ff42b49941f4c11df1434f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "d78a1d347bc44a82b3cae400b801bf9d" is_pre_election: true
I20240506 11:40:24.846309 14925 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84bc0a6756ff42b49941f4c11df1434f in term 2.
I20240506 11:40:24.847385 15226 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513)
I20240506 11:40:24.852083 14990 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038, 84bc0a6756ff42b49941f4c11df1434f, d78a1d347bc44a82b3cae400b801bf9d; no voters:
I20240506 11:40:24.852480 14860 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038, 84bc0a6756ff42b49941f4c11df1434f, d78a1d347bc44a82b3cae400b801bf9d; no voters:
I20240506 11:40:24.852519 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "680e178c01b14b439b3fe7e7d2a33038" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "d78a1d347bc44a82b3cae400b801bf9d" is_pre_election: true
I20240506 11:40:24.852732 14925 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680e178c01b14b439b3fe7e7d2a33038 in term 2.
I20240506 11:40:24.852860 15224 raft_consensus.cc:2781] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 11:40:24.852979 15224 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a7bb6d01d54afdaa32a7aa261cb623)
I20240506 11:40:24.853086 15224 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 2 FOLLOWER]: Advancing to term 3
I20240506 11:40:24.853224 14599 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038, 84bc0a6756ff42b49941f4c11df1434f, d78a1d347bc44a82b3cae400b801bf9d; no voters:
I20240506 11:40:24.853492 15225 raft_consensus.cc:2781] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 11:40:24.853549 15226 raft_consensus.cc:2781] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240506 11:40:24.853596 15225 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a7bb6d01d54afdaa32a7aa261cb623)
I20240506 11:40:24.853674 15225 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 2 FOLLOWER]: Advancing to term 3
I20240506 11:40:24.853695 15226 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Starting leader election (detected failure of leader d0a7bb6d01d54afdaa32a7aa261cb623)
I20240506 11:40:24.853799 15226 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 2 FOLLOWER]: Advancing to term 3
W20240506 11:40:24.854045 14992 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
W20240506 11:40:24.854718 14990 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111)
W20240506 11:40:24.855649 14860 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111)
I20240506 11:40:24.855791 15224 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
W20240506 11:40:24.855947 14862 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:24.856238 15225 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:24.856403 15226 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:24.856729 15224 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 3 election: Requested vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
W20240506 11:40:24.856804 14600 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111)
I20240506 11:40:24.857201 15055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f"
I20240506 11:40:24.857421 15055 raft_consensus.cc:2379] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d78a1d347bc44a82b3cae400b801bf9d in current term 3: Already voted for candidate 84bc0a6756ff42b49941f4c11df1434f in this term.
W20240506 11:40:24.857628 14602 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:24.858071 14665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038"
I20240506 11:40:24.858291 14665 raft_consensus.cc:2379] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d78a1d347bc44a82b3cae400b801bf9d in current term 3: Already voted for candidate 680e178c01b14b439b3fe7e7d2a33038 in this term.
W20240506 11:40:24.858697 14862 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:24.858913 15226 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 3 election: Requested vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513)
I20240506 11:40:24.859261 15055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "680e178c01b14b439b3fe7e7d2a33038" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f"
I20240506 11:40:24.859478 15055 raft_consensus.cc:2379] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 680e178c01b14b439b3fe7e7d2a33038 in current term 3: Already voted for candidate 84bc0a6756ff42b49941f4c11df1434f in this term.
I20240506 11:40:24.859452 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "84bc0a6756ff42b49941f4c11df1434f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "d78a1d347bc44a82b3cae400b801bf9d"
I20240506 11:40:24.859637 14925 raft_consensus.cc:2379] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bc0a6756ff42b49941f4c11df1434f in current term 3: Already voted for candidate d78a1d347bc44a82b3cae400b801bf9d in this term.
I20240506 11:40:24.859920 15225 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 3 election: Requested vote from peers d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
I20240506 11:40:24.860180 14665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "84bc0a6756ff42b49941f4c11df1434f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038"
W20240506 11:40:24.860309 14992 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
I20240506 11:40:24.860379 14665 raft_consensus.cc:2379] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 84bc0a6756ff42b49941f4c11df1434f in current term 3: Already voted for candidate 680e178c01b14b439b3fe7e7d2a33038 in this term.
I20240506 11:40:24.860649 14925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "680e178c01b14b439b3fe7e7d2a33038" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "d78a1d347bc44a82b3cae400b801bf9d"
I20240506 11:40:24.860826 14925 raft_consensus.cc:2379] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 680e178c01b14b439b3fe7e7d2a33038 in current term 3: Already voted for candidate d78a1d347bc44a82b3cae400b801bf9d in this term.
I20240506 11:40:24.861063 14860 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d78a1d347bc44a82b3cae400b801bf9d; no voters: 680e178c01b14b439b3fe7e7d2a33038, 84bc0a6756ff42b49941f4c11df1434f, c1922e4786c34109bd610416f0f99872
W20240506 11:40:24.861140 14602 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
W20240506 11:40:24.861318 14600 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111)
W20240506 11:40:24.861305 14990 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111)
I20240506 11:40:24.861438 15224 raft_consensus.cc:2726] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 11:40:24.861425 14990 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 84bc0a6756ff42b49941f4c11df1434f; no voters: c1922e4786c34109bd610416f0f99872, d0a7bb6d01d54afdaa32a7aa261cb623, d78a1d347bc44a82b3cae400b801bf9d
W20240506 11:40:24.861418 14860 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111)
I20240506 11:40:24.861512 14600 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038; no voters: 84bc0a6756ff42b49941f4c11df1434f, c1922e4786c34109bd610416f0f99872, d0a7bb6d01d54afdaa32a7aa261cb623
I20240506 11:40:24.861757 15226 raft_consensus.cc:2726] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 11:40:24.861838 15225 raft_consensus.cc:2726] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 11:40:24.962792 10422 test_util.cc:268] Using random seed: -1996199231
I20240506 11:40:25.569978 10422 test_util.cc:268] Using random seed: -1995592033
I20240506 11:40:26.277500 10422 test_util.cc:268] Using random seed: -1994884539
I20240506 11:40:26.402187 15239 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:40:26.402357 15239 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:26.403167 15239 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
I20240506 11:40:26.403613 14665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038" is_pre_election: true
I20240506 11:40:26.403887 14665 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78a1d347bc44a82b3cae400b801bf9d in term 3.
I20240506 11:40:26.403817 15055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f" is_pre_election: true
I20240506 11:40:26.404105 15055 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d78a1d347bc44a82b3cae400b801bf9d in term 3.
I20240506 11:40:26.405481 14860 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038, 84bc0a6756ff42b49941f4c11df1434f, d78a1d347bc44a82b3cae400b801bf9d; no voters:
I20240506 11:40:26.405803 15239 raft_consensus.cc:2781] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 11:40:26.405933 15239 raft_consensus.cc:482] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:40:26.406025 15239 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 3 FOLLOWER]: Advancing to term 4
W20240506 11:40:26.406575 14862 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
W20240506 11:40:26.406929 14860 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111)
I20240506 11:40:26.408210 15239 raft_consensus.cc:504] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:26.409090 15239 leader_election.cc:288] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 4 election: Requested vote from peers 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181), d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513), c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109), 84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953)
I20240506 11:40:26.409140 14665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "680e178c01b14b439b3fe7e7d2a33038"
I20240506 11:40:26.409317 14665 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 3 FOLLOWER]: Advancing to term 4
I20240506 11:40:26.409332 15055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb8cbb86424c2d9ec72dc29b001ab0" candidate_uuid: "d78a1d347bc44a82b3cae400b801bf9d" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "84bc0a6756ff42b49941f4c11df1434f"
I20240506 11:40:26.409539 15055 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 3 FOLLOWER]: Advancing to term 4
I20240506 11:40:26.411127 14665 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d78a1d347bc44a82b3cae400b801bf9d in term 4.
I20240506 11:40:26.411350 15055 raft_consensus.cc:2454] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d78a1d347bc44a82b3cae400b801bf9d in term 4.
I20240506 11:40:26.411705 14861 leader_election.cc:302] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 680e178c01b14b439b3fe7e7d2a33038, 84bc0a6756ff42b49941f4c11df1434f, d78a1d347bc44a82b3cae400b801bf9d; no voters:
I20240506 11:40:26.412011 15239 raft_consensus.cc:2781] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 4 FOLLOWER]: Leader election won for term 4
I20240506 11:40:26.412518 15239 raft_consensus.cc:686] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [term 4 LEADER]: Becoming Leader. State: Replica: d78a1d347bc44a82b3cae400b801bf9d, State: Running, Role: LEADER
W20240506 11:40:26.412798 14862 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111)
W20240506 11:40:26.413050 14860 leader_election.cc:334] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111)
I20240506 11:40:26.413079 15239 consensus_queue.cc:237] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:26.416443 14393 catalog_manager.cc:5484] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d reported cstate change: term changed from 2 to 4, leader changed from d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129) to d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132). New cstate: current_term: 4 leader_uuid: "d78a1d347bc44a82b3cae400b801bf9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } health_report { overall_health: UNKNOWN } } }
W20240506 11:40:26.827366 14862 consensus_peers.cc:485] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d -> Peer c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): Couldn't send request to peer c1922e4786c34109bd610416f0f99872. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.131:46109: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:40:26.837582 14665 raft_consensus.cc:1261] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 680e178c01b14b439b3fe7e7d2a33038 [term 4 FOLLOWER]: Refusing update from remote peer d78a1d347bc44a82b3cae400b801bf9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240506 11:40:26.838145 15239 consensus_queue.cc:1035] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240506 11:40:26.907610 15055 raft_consensus.cc:1261] T f8cb8cbb86424c2d9ec72dc29b001ab0 P 84bc0a6756ff42b49941f4c11df1434f [term 4 FOLLOWER]: Refusing update from remote peer d78a1d347bc44a82b3cae400b801bf9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240506 11:40:26.908190 15239 consensus_queue.cc:1035] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240506 11:40:26.914661 14860 consensus_peers.cc:485] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d -> Peer d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): Couldn't send request to peer d0a7bb6d01d54afdaa32a7aa261cb623. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:40:27.084664 10422 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (3 alive)...
W20240506 11:40:27.087848 15105 meta_cache.cc:302] tablet f8cb8cbb86424c2d9ec72dc29b001ab0: replica d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513) has failed: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111)
W20240506 11:40:27.091161 14626 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45136: Illegal state: replica 680e178c01b14b439b3fe7e7d2a33038 is not leader of this config: current role FOLLOWER
I20240506 11:40:27.128190 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:33513
--local_ip_for_outbound_sockets=127.10.45.129
--tserver_master_addrs=127.10.45.190:39273
--webserver_port=35333
--webserver_interface=127.10.45.129
--builtin_ntp_servers=127.10.45.148:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20240506 11:40:27.428042 15265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:27.428313 15265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:27.428396 15265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:27.435324 15265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:27.435467 15265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:40:27.443222 15265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:33513
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=35333
--tserver_master_addrs=127.10.45.190:39273
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:27.444348 15265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:27.445896 15265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:27.457811 15271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:27.458694 15273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:27.459497 15270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:27.460373 15265 server_base.cc:1034] running on GCE node
I20240506 11:40:27.460863 15265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:27.461640 15265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:27.462844 15265 hybrid_clock.cc:648] HybridClock initialized: now 1714995627462788 us; error 35 us; skew 500 ppm
I20240506 11:40:27.463130 15265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:27.470387 15265 webserver.cc:460] Webserver started at http://127.10.45.129:35333/ using document root <none> and password file <none>
I20240506 11:40:27.471056 15265 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:27.471176 15265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:27.476064 15265 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240506 11:40:27.478528 15279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:27.479780 15265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240506 11:40:27.479913 15265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d0a7bb6d01d54afdaa32a7aa261cb623"
format_stamp: "Formatted at 2024-05-06 11:40:19 on dist-test-slave-7l0g"
I20240506 11:40:27.480664 15265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:27.496690 15265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:27.497372 15265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:27.497606 15265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:27.498407 15265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:27.500175 15286 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240506 11:40:27.504714 15265 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 11:40:27.504802 15265 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20240506 11:40:27.504877 15265 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 11:40:27.507480 15265 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 11:40:27.507570 15265 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20240506 11:40:27.507912 15286 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Bootstrap starting.
I20240506 11:40:27.534782 15286 log.cc:824] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:27.542074 15265 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:33513
I20240506 11:40:27.542121 15393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:33513 every 8 connection(s)
I20240506 11:40:27.543337 15265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:40:27.546197 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 15265
I20240506 11:40:27.547147 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:46109
--local_ip_for_outbound_sockets=127.10.45.131
--tserver_master_addrs=127.10.45.190:39273
--webserver_port=32947
--webserver_interface=127.10.45.131
--builtin_ntp_servers=127.10.45.148:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20240506 11:40:27.564270 15286 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 11:40:27.564746 15286 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Bootstrap complete.
I20240506 11:40:27.566131 15286 ts_tablet_manager.cc:1367] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Time spent bootstrapping tablet: real 0.059s user 0.042s sys 0.012s
I20240506 11:40:27.570029 15394 heartbeater.cc:347] Connected to a master server at 127.10.45.190:39273
I20240506 11:40:27.570348 15394 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:27.571348 15394 heartbeater.cc:511] Master 127.10.45.190:39273 requested a full tablet report, sending...
I20240506 11:40:27.572093 15286 raft_consensus.cc:348] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:27.573180 15286 raft_consensus.cc:729] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a7bb6d01d54afdaa32a7aa261cb623, State: Initialized, Role: FOLLOWER
I20240506 11:40:27.573856 15286 consensus_queue.cc:260] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:27.574149 14395 ts_manager.cc:194] Re-registered known tserver with Master: d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513)
I20240506 11:40:27.575865 14395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:42815
I20240506 11:40:27.582675 15394 heartbeater.cc:503] Master 127.10.45.190:39273 was elected leader, sending a full tablet report...
I20240506 11:40:27.583129 15286 ts_tablet_manager.cc:1398] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623: Time spent starting tablet: real 0.017s user 0.011s sys 0.009s
I20240506 11:40:27.762140 15348 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d0a7bb6d01d54afdaa32a7aa261cb623 [term 2 FOLLOWER]: Advancing to term 4
W20240506 11:40:27.863833 15398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:27.864065 15398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:27.864142 15398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:27.870543 15398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:27.870695 15398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:40:27.878527 15398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:46109
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=32947
--tserver_master_addrs=127.10.45.190:39273
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:27.879717 15398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:27.881229 15398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:27.892158 15410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:27.893024 15411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:27.893755 15413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:27.894538 15398 server_base.cc:1034] running on GCE node
I20240506 11:40:27.895061 15398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:27.895793 15398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:27.896991 15398 hybrid_clock.cc:648] HybridClock initialized: now 1714995627896943 us; error 26 us; skew 500 ppm
I20240506 11:40:27.897238 15398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:27.899261 15398 webserver.cc:460] Webserver started at http://127.10.45.131:32947/ using document root <none> and password file <none>
I20240506 11:40:27.899888 15398 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:27.900012 15398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:27.904807 15398 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20240506 11:40:27.907205 15420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:27.908269 15398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240506 11:40:27.908416 15398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c1922e4786c34109bd610416f0f99872"
format_stamp: "Formatted at 2024-05-06 11:40:20 on dist-test-slave-7l0g"
I20240506 11:40:27.909111 15398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:27.922050 15398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:27.922744 15398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:27.923013 15398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:27.923914 15398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:27.925809 15427 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240506 11:40:27.930353 15398 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 11:40:27.930460 15398 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20240506 11:40:27.930569 15398 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 11:40:27.933240 15398 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 11:40:27.933321 15398 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20240506 11:40:27.933624 15427 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Bootstrap starting.
I20240506 11:40:27.954716 15427 log.cc:824] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:27.971617 15398 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:46109
I20240506 11:40:27.971669 15534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:46109 every 8 connection(s)
I20240506 11:40:27.972874 15427 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 11:40:27.973117 15398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:40:27.973435 15427 tablet_bootstrap.cc:492] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Bootstrap complete.
I20240506 11:40:27.974481 15427 ts_tablet_manager.cc:1367] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Time spent bootstrapping tablet: real 0.041s user 0.019s sys 0.018s
I20240506 11:40:27.980312 15427 raft_consensus.cc:348] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:27.981124 15427 raft_consensus.cc:729] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1922e4786c34109bd610416f0f99872, State: Initialized, Role: FOLLOWER
I20240506 11:40:27.981755 15427 consensus_queue.cc:260] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680e178c01b14b439b3fe7e7d2a33038" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 46181 } } peers { permanent_uuid: "d0a7bb6d01d54afdaa32a7aa261cb623" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33513 } } peers { permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 } } peers { permanent_uuid: "d78a1d347bc44a82b3cae400b801bf9d" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 45513 } } peers { permanent_uuid: "84bc0a6756ff42b49941f4c11df1434f" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 43953 } }
I20240506 11:40:27.982759 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 15398
I20240506 11:40:27.989809 15427 ts_tablet_manager.cc:1398] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872: Time spent starting tablet: real 0.015s user 0.014s sys 0.003s
I20240506 11:40:27.996762 15535 heartbeater.cc:347] Connected to a master server at 127.10.45.190:39273
I20240506 11:40:27.997066 15535 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:27.997962 15535 heartbeater.cc:511] Master 127.10.45.190:39273 requested a full tablet report, sending...
I20240506 11:40:28.000347 14393 ts_manager.cc:194] Re-registered known tserver with Master: c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109)
I20240506 11:40:28.001492 14393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:36697
I20240506 11:40:28.004012 15535 heartbeater.cc:503] Master 127.10.45.190:39273 was elected leader, sending a full tablet report...
I20240506 11:40:28.077649 14646 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:40:28.082415 15036 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:40:28.104184 15328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:40:28.109643 15469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:40:28.110569 14906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
13bfa32bcf7a4f9787c1278d0e7d61da | 127.10.45.190:39273 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:43843 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
680e178c01b14b439b3fe7e7d2a33038 | 127.10.45.130:46181 | HEALTHY | <none> | 0 | 0
84bc0a6756ff42b49941f4c11df1434f | 127.10.45.133:43953 | HEALTHY | <none> | 0 | 0
c1922e4786c34109bd610416f0f99872 | 127.10.45.131:46109 | HEALTHY | <none> | 0 | 0
d0a7bb6d01d54afdaa32a7aa261cb623 | 127.10.45.129:33513 | HEALTHY | <none> | 0 | 0
d78a1d347bc44a82b3cae400b801bf9d | 127.10.45.132:45513 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.10.45.129 | experimental | 127.10.45.129:33513
local_ip_for_outbound_sockets | 127.10.45.130 | experimental | 127.10.45.130:46181
local_ip_for_outbound_sockets | 127.10.45.131 | experimental | 127.10.45.131:46109
local_ip_for_outbound_sockets | 127.10.45.132 | experimental | 127.10.45.132:45513
local_ip_for_outbound_sockets | 127.10.45.133 | experimental | 127.10.45.133:43953
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.45.129:33513
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.45.130:46181
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.45.131:46109
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.10.45.132:45513
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.10.45.133:43953
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:43843 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
Tablet f8cb8cbb86424c2d9ec72dc29b001ab0 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181): RUNNING
d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): RUNNING
c1922e4786c34109bd610416f0f99872 (127.10.45.131:46109): RUNNING
d78a1d347bc44a82b3cae400b801bf9d (127.10.45.132:45513): RUNNING [LEADER]
84bc0a6756ff42b49941f4c11df1434f (127.10.45.133:43953): RUNNING
All reported replicas are:
A = 680e178c01b14b439b3fe7e7d2a33038
B = d0a7bb6d01d54afdaa32a7aa261cb623
C = c1922e4786c34109bd610416f0f99872
D = d78a1d347bc44a82b3cae400b801bf9d
E = 84bc0a6756ff42b49941f4c11df1434f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------------------+--------------+--------------+------------
master | A B C D* E | | | Yes
A | A B C D* E | 4 | -1 | Yes
B | A B C D* E | 4 | -1 | Yes
C | A B C D E | 1 | -1 | Yes
D | A B C D* E | 4 | -1 | Yes
E | A B C D* E | 4 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 5 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20240506 11:40:28.159307 10422 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20240506 11:40:28.398372 15490 raft_consensus.cc:3037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 1 FOLLOWER]: Advancing to term 4
I20240506 11:40:28.401129 15328 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:40:28.402092 15036 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 11:40:28.404636 15490 raft_consensus.cc:1261] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 4 FOLLOWER]: Refusing update from remote peer d78a1d347bc44a82b3cae400b801bf9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240506 11:40:28.405508 15569 consensus_queue.cc:1037] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "c1922e4786c34109bd610416f0f99872" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 6, Time since last communication: 0.000s
I20240506 11:40:28.407282 14906 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240506 11:40:28.407431 14646 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 11:40:28.448273 15490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.45.132:49129 (request call id 1) took 41 ms (client timeout 50 ms). Trace:
I20240506 11:40:28.450989 15469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 11:40:28.448455 15490 rpcz_store.cc:269] 0506 11:40:28.406262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:28.406324 (+ 62us) service_pool.cc:225] Handling call
0506 11:40:28.406690 (+ 366us) raft_consensus.cc:1457] Updating replica for 12 ops
0506 11:40:28.407643 (+ 953us) raft_consensus.cc:1520] Early marking committed up to index 0
0506 11:40:28.407651 (+ 8us) raft_consensus.cc:1525] Triggering prepare for 12 ops
0506 11:40:28.408578 (+ 927us) write_op.cc:270] Start()
0506 11:40:28.408632 (+ 54us) write_op.cc:276] Timestamp: P: 1714995623275930 usec, L: 0
0506 11:40:28.410166 (+ 1534us) write_op.cc:270] Start()
0506 11:40:28.410192 (+ 26us) write_op.cc:276] Timestamp: P: 1714995623295793 usec, L: 0
0506 11:40:28.410374 (+ 182us) write_op.cc:270] Start()
0506 11:40:28.410405 (+ 31us) write_op.cc:276] Timestamp: P: 1714995623302877 usec, L: 0
0506 11:40:28.410570 (+ 165us) write_op.cc:270] Start()
0506 11:40:28.410589 (+ 19us) write_op.cc:276] Timestamp: P: 1714995623309471 usec, L: 0
0506 11:40:28.410724 (+ 135us) write_op.cc:270] Start()
0506 11:40:28.410858 (+ 134us) write_op.cc:276] Timestamp: P: 1714995623315918 usec, L: 0
0506 11:40:28.411050 (+ 192us) write_op.cc:270] Start()
0506 11:40:28.411073 (+ 23us) write_op.cc:276] Timestamp: P: 1714995627101271 usec, L: 0
0506 11:40:28.411230 (+ 157us) write_op.cc:270] Start()
0506 11:40:28.411303 (+ 73us) write_op.cc:276] Timestamp: P: 1714995627108124 usec, L: 0
0506 11:40:28.411441 (+ 138us) write_op.cc:270] Start()
0506 11:40:28.411459 (+ 18us) write_op.cc:276] Timestamp: P: 1714995627113592 usec, L: 0
0506 11:40:28.411594 (+ 135us) write_op.cc:270] Start()
0506 11:40:28.411612 (+ 18us) write_op.cc:276] Timestamp: P: 1714995627118534 usec, L: 0
0506 11:40:28.411724 (+ 112us) write_op.cc:270] Start()
0506 11:40:28.411742 (+ 18us) write_op.cc:276] Timestamp: P: 1714995627123679 usec, L: 0
0506 11:40:28.412415 (+ 673us) log.cc:842] Serialized 6732 byte log entry
0506 11:40:28.419265 (+ 6850us) raft_consensus.cc:1631] Marking committed up to 18
0506 11:40:28.419439 (+ 174us) log.cc:842] Serialized 14 byte log entry
0506 11:40:28.443229 (+ 23790us) log.cc:842] Serialized 14 byte log entry
0506 11:40:28.447423 (+ 4194us) raft_consensus.cc:1683] Filling consensus response to leader.
0506 11:40:28.448118 (+ 695us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0506 11:40:28.448142 (+ 24us) raft_consensus.cc:1671] finished
0506 11:40:28.448146 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0506 11:40:28.448204 (+ 58us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:28.410068 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f8cb8cbb86424c2d9ec72dc29b001ab0
0506 11:40:28.410211 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:28.410216 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:40:28.410218 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:28.410584 (+ 366us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:28.410602 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:28.410613 (+ 11us) tablet.cc:683] Acquiring locks for 10 operations
0506 11:40:28.410735 (+ 122us) tablet.cc:699] Row locks acquired
0506 11:40:28.410738 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:40:28.419649 (+ 8911us) op_driver.cc:444] REPLICATION: finished
0506 11:40:28.439626 (+ 19977us) write_op.cc:301] APPLY: starting
0506 11:40:28.439648 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:28.439981 (+ 333us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:28.439984 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:28.440326 (+ 342us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:28.440557 (+ 231us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:28.440582 (+ 25us) write_op.cc:312] APPLY: finished
0506 11:40:28.440768 (+ 186us) log.cc:842] Serialized 76 byte log entry
0506 11:40:28.440890 (+ 122us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.440917 (+ 27us) write_op.cc:454] Released schema lock
0506 11:40:28.440930 (+ 13us) write_op.cc:341] FINISH: Updating metrics
0506 11:40:28.441179 (+ 249us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.441184 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:40:28.410844 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f8cb8cbb86424c2d9ec72dc29b001ab0
0506 11:40:28.410922 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:28.410926 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:28.410928 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:28.411337 (+ 409us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:28.411364 (+ 27us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:28.411368 (+ 4us) tablet.cc:683] Acquiring locks for 10 operations
0506 11:40:28.411467 (+ 99us) tablet.cc:699] Row locks acquired
0506 11:40:28.411469 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:28.439833 (+ 28364us) op_driver.cc:444] REPLICATION: finished
0506 11:40:28.440323 (+ 490us) write_op.cc:301] APPLY: starting
0506 11:40:28.440337 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:28.440457 (+ 120us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:28.440461 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:28.441182 (+ 721us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:28.441287 (+ 105us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:28.441296 (+ 9us) write_op.cc:312] APPLY: finished
0506 11:40:28.441434 (+ 138us) log.cc:842] Serialized 76 byte log entry
0506 11:40:28.441672 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.441695 (+ 23us) write_op.cc:454] Released schema lock
0506 11:40:28.441702 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 11:40:28.441766 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.441770 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:40:28.411545 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f8cb8cbb86424c2d9ec72dc29b001ab0
0506 11:40:28.411630 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:28.411634 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:28.411636 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:28.411770 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:28.411781 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:28.411785 (+ 4us) tablet.cc:683] Acquiring locks for 10 operations
0506 11:40:28.411855 (+ 70us) tablet.cc:699] Row locks acquired
0506 11:40:28.411857 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:28.440223 (+ 28366us) op_driver.cc:444] REPLICATION: finished
0506 11:40:28.441316 (+ 1093us) write_op.cc:301] APPLY: starting
0506 11:40:28.441326 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:28.441375 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:28.441377 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:28.441610 (+ 233us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:28.441668 (+ 58us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:28.441676 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:40:28.441752 (+ 76us) log.cc:842] Serialized 76 byte log entry
0506 11:40:28.441816 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.441836 (+ 20us) write_op.cc:454] Released schema lock
0506 11:40:28.441842 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:40:28.441909 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.441914 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:40:28.411922 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f8cb8cbb86424c2d9ec72dc29b001ab0
0506 11:40:28.411987 (+ 65us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:28.411991 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:28.411993 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:28.412111 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:28.412121 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:28.412125 (+ 4us) tablet.cc:683] Acquiring locks for 10 operations
0506 11:40:28.412214 (+ 89us) tablet.cc:699] Row locks acquired
0506 11:40:28.412216 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:28.443043 (+ 30827us) op_driver.cc:444] REPLICATION: finished
0506 11:40:28.443681 (+ 638us) write_op.cc:301] APPLY: starting
0506 11:40:28.443692 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:28.443746 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:28.443748 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:28.443922 (+ 174us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:28.443967 (+ 45us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:28.443974 (+ 7us) write_op.cc:312] APPLY: finished
0506 11:40:28.444038 (+ 64us) log.cc:842] Serialized 76 byte log entry
0506 11:40:28.444211 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.444228 (+ 17us) write_op.cc:454] Released schema lock
0506 11:40:28.444234 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:40:28.444280 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.444283 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:40:28.412297 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f8cb8cbb86424c2d9ec72dc29b001ab0
0506 11:40:28.412402 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:28.412406 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:28.412408 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:28.412536 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:28.412546 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:28.412552 (+ 6us) tablet.cc:683] Acquiring locks for 10 operations
0506 11:40:28.412727 (+ 175us) tablet.cc:699] Row locks acquired
0506 11:40:28.412731 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:40:28.443102 (+ 30371us) op_driver.cc:444] REPLICATION: finished
0506 11:40:28.444348 (+ 1246us) write_op.cc:301] APPLY: starting
0506 11:40:28.444359 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:28.444397 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:28.444400 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:28.444575 (+ 175us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:28.444620 (+ 45us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:28.444626 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:40:28.444688 (+ 62us) log.cc:842] Serialized 76 byte log entry
0506 11:40:28.444842 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.444858 (+ 16us) write_op.cc:454] Released schema lock
0506 11:40:28.444864 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:40:28.444910 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.444914 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:40:28.412829 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f8cb8cbb86424c2d9ec72dc29b001ab0
0506 11:40:28.412927 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:28.412934 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:40:28.412937 (+ 3us) tablet.cc:660] Decoding operations
0506 11:40:28.413134 (+ 197us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:28.413149 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:28.413157 (+ 8us) tablet.cc:683] Acquiring locks for 10 operations
0506 11:40:28.413479 (+ 322us) tablet.cc:699] Row locks acquired
0506 11:40:28.413483 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:40:28.443498 (+ 30015us) op_driver.cc:444] REPLICATION: finished
0506 11:40:28.444978 (+ 1480us) write_op.cc:301] APPLY: starting
0506 11:40:28.444987 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:28.445029 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:28.445033 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:28.445260 (+ 227us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:28.445308 (+ 48us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:28.445316 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:40:28.445373 (+ 57us) log.cc:842] Serialized 76 byte log entry
0506 11:40:28.445538 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.445554 (+ 16us) write_op.cc:454] Released schema lock
0506 11:40:28.445560 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0506 11:40:28.445605 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.445609 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:40:28.413571 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f8cb8cbb86424c2d9ec72dc29b001ab0
0506 11:40:28.413707 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:28.413714 (+ 7us) write_op.cc:435] Acquired schema lock
0506 11:40:28.413717 (+ 3us) tablet.cc:660] Decoding operations
0506 11:40:28.413897 (+ 180us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:28.413911 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:28.413918 (+ 7us) tablet.cc:683] Acquiring locks for 10 operations
0506 11:40:28.414055 (+ 137us) tablet.cc:699] Row locks acquired
0506 11:40:28.414058 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:40:28.443535 (+ 29477us) op_driver.cc:444] REPLICATION: finished
0506 11:40:28.445675 (+ 2140us) write_op.cc:301] APPLY: starting
0506 11:40:28.445684 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:28.445723 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:28.445725 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:28.445895 (+ 170us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:28.445945 (+ 50us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:28.445951 (+ 6us) write_op.cc:312] APPLY: finished
0506 11:40:28.446013 (+ 62us) log.cc:842] Serialized 76 byte log entry
0506 11:40:28.446174 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.446190 (+ 16us) write_op.cc:454] Released schema lock
0506 11:40:28.446197 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0506 11:40:28.446245 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.446249 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0506 11:40:28.414142 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f8cb8cbb86424c2d9ec72dc29b001ab0
0506 11:40:28.414228 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:28.414234 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:40:28.414238 (+ 4us) tablet.cc:660] Decoding operations
0506 11:40:28.414422 (+ 184us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:28.414435 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:28.414441 (+ 6us) tablet.cc:683] Acquiring locks for 10 operations
0506 11:40:28.414555 (+ 114us) tablet.cc:699] Row locks acquired
0506 11:40:28.414559 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:40:28.446389 (+ 31830us) op_driver.cc:444] REPLICATION: finished
0506 11:40:28.446568 (+ 179us) write_op.cc:301] APPLY: starting
0506 11:40:28.446581 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:28.446661 (+ 80us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:28.446663 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:28.447002 (+ 339us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:28.447085 (+ 83us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:28.447094 (+ 9us) write_op.cc:312] APPLY: finished
0506 11:40:28.447179 (+ 85us) log.cc:842] Serialized 76 byte log entry
0506 11:40:28.447228 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:28.447244 (+ 16us) write_op.cc:454] Released schema lock
0506 11:40:28.447250 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0506 11:40:28.414647 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f8cb8cbb86424c2d9ec72dc29b001ab0
0506 11:40:28.414726 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:28.414732 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:40:28.414735 (+ 3us) tablet.cc:660] Decoding operations
0506 11:40:28.414897 (+ 162us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:28.414909 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:28.414916 (+ 7us) tablet.cc:683] Acquiring locks for 10 operations
0506 11:40:28.415028 (+ 112us) tablet.cc:699] Row locks acquired
0506 11:40:28.415032 (+ 4us) write_op.cc:260] PREPARE: finished
0506 11:40:28.446489 (+ 31457us) op_driver.cc:444] REPLICATION: finished
0506 11:40:28.447385 (+ 896us) write_op.cc:301] APPLY: starting
Related trace 'op':
0506 11:40:28.415125 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f8cb8cbb86424c2d9ec72dc29b001ab0
0506 11:40:28.415218 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:28.415224 (+ 6us) write_op.cc:435] Acquired schema lock
0506 11:40:28.415227 (+ 3us) tablet.cc:660] Decoding operations
0506 11:40:28.415429 (+ 202us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:28.415442 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:28.415449 (+ 7us) tablet.cc:683] Acquiring locks for 10 operations
0506 11:40:28.415561 (+ 112us) tablet.cc:699] Row locks acquired
0506 11:40:28.415564 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:40:28.447369 (+ 31805us) op_driver.cc:444] REPLICATION: finished
0506 11:40:28.447424 (+ 55us) write_op.cc:301] APPLY: starting
0506 11:40:28.447430 (+ 6us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:28.447607 (+ 177us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:28.447611 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
Metrics: {"prepare.queue_time_us":149,"prepare.run_cpu_time_us":14,"prepare.run_wall_time_us":15,"raft.queue_time_us":148,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"thread_start_us":6677,"threads_started":1,"wal-append.queue_time_us":6271,"child_traces":[["op",{"apply.queue_time_us":19883,"apply.run_cpu_time_us":1663,"apply.run_wall_time_us":1664,"num_ops":10,"partition_lock_wait_us":7,"prepare.queue_time_us":326,"prepare.run_cpu_time_us":732,"prepare.run_wall_time_us":762,"replication_time_us":11055,"thread_start_us":20324,"threads_started":2}],["op",{"apply.queue_time_us":415,"apply.run_cpu_time_us":1087,"apply.run_wall_time_us":1628,"num_ops":10,"partition_lock_wait_us":17,"prepare.queue_time_us":583,"prepare.run_cpu_time_us":434,"prepare.run_wall_time_us":668,"replication_time_us":29648,"thread_start_us":286,"threads_started":1}],["op",{"apply.queue_time_us":1054,"apply.run_cpu_time_us":642,"apply.run_wall_time_us":641,"mutex_wait_us":45,"num_ops":10,"partition_lock_wait_us":3,"prepare.queue_time_us":1071,"prepare.run_cpu_time_us":350,"prepare.run_wall_time_us":349,"replication_time_us":29834,"thread_start_us":2684,"threads_started":1}],["op",{"apply.queue_time_us":598,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":642,"num_ops":10,"partition_lock_wait_us":2,"prepare.queue_time_us":1281,"prepare.run_cpu_time_us":331,"prepare.run_wall_time_us":330,"replication_time_us":32460}],["op",{"apply.queue_time_us":1223,"apply.run_cpu_time_us":495,"apply.run_wall_time_us":603,"num_ops":10,"partition_lock_wait_us":3,"prepare.queue_time_us":1381,"prepare.run_cpu_time_us":444,"prepare.run_wall_time_us":487,"replication_time_us":32369}],["op",{"apply.queue_time_us":1454,"apply.run_cpu_time_us":553,"apply.run_wall_time_us":669,"num_ops":10,"partition_lock_wait_us":3,"prepare.queue_time_us":1680,"prepare.run_cpu_time_us":531,"prepare.run_wall_time_us":704,"replication_time_us":32437}],["op",{"apply.queue_time_us":2117,"apply.run_cpu_time_us":512,"apply.run_wall_time_us":626,"num_ops":10,"partition_lock_wait_us":3,"prepare.queue_time_us":2202,"prepare.run_cpu_time_us":508,"prepare.run_wall_time_us":533,"replication_time_us":32298,"thread_start_us":2801,"threads_started":1}],["op",{"apply.queue_time_us":98,"num_ops":10,"partition_lock_wait_us":2,"prepare.queue_time_us":2625,"prepare.run_cpu_time_us":472,"prepare.run_wall_time_us":472,"replication_time_us":34938}],["op",{"apply.queue_time_us":838,"mutex_wait_us":801,"num_ops":10,"partition_lock_wait_us":2,"prepare.queue_time_us":2980,"prepare.run_cpu_time_us":429,"prepare.run_wall_time_us":428,"replication_time_us":34870}],["op",{"apply.queue_time_us":26,"num_ops":10,"partition_lock_wait_us":3,"prepare.queue_time_us":3322,"prepare.run_cpu_time_us":495,"prepare.run_wall_time_us":494,"replication_time_us":35631}]]}
W20240506 11:40:28.459098 14862 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.10.45.131:46109 timed out after 0.050s (SENT) [suppressed 13 similar messages]
I20240506 11:40:28.462261 15490 raft_consensus.cc:1203] T f8cb8cbb86424c2d9ec72dc29b001ab0 P c1922e4786c34109bd610416f0f99872 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[2.7-4.18] Dedup: 4.18->[]
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
13bfa32bcf7a4f9787c1278d0e7d61da | 127.10.45.190:39273 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:43843 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
680e178c01b14b439b3fe7e7d2a33038 | 127.10.45.130:46181 | HEALTHY | <none> | 0 | 0
84bc0a6756ff42b49941f4c11df1434f | 127.10.45.133:43953 | HEALTHY | <none> | 0 | 0
c1922e4786c34109bd610416f0f99872 | 127.10.45.131:46109 | HEALTHY | <none> | 0 | 0
d0a7bb6d01d54afdaa32a7aa261cb623 | 127.10.45.129:33513 | HEALTHY | <none> | 0 | 0
d78a1d347bc44a82b3cae400b801bf9d | 127.10.45.132:45513 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.10.45.129 | experimental | 127.10.45.129:33513
local_ip_for_outbound_sockets | 127.10.45.130 | experimental | 127.10.45.130:46181
local_ip_for_outbound_sockets | 127.10.45.131 | experimental | 127.10.45.131:46109
local_ip_for_outbound_sockets | 127.10.45.132 | experimental | 127.10.45.132:45513
local_ip_for_outbound_sockets | 127.10.45.133 | experimental | 127.10.45.133:43953
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.45.129:33513
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.45.130:46181
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.45.131:46109
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.10.45.132:45513
server_dump_info_path | /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.10.45.133:43953
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.10.45.148:43843 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240506 11:40:28.487393 10422 log_verifier.cc:126] Checking tablet f8cb8cbb86424c2d9ec72dc29b001ab0
I20240506 11:40:28.534214 10422 log_verifier.cc:177] Verified matching terms for 18 ops in tablet f8cb8cbb86424c2d9ec72dc29b001ab0
I20240506 11:40:28.707486 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 15265
I20240506 11:40:28.713848 15389 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:28.797353 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 15265
I20240506 11:40:28.813833 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 14584
I20240506 11:40:28.814821 14706 generic_service.cc:196] Checking for leaks (request via RPC)
W20240506 11:40:28.850538 14860 consensus_peers.cc:485] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d -> Peer d0a7bb6d01d54afdaa32a7aa261cb623 (127.10.45.129:33513): Couldn't send request to peer d0a7bb6d01d54afdaa32a7aa261cb623. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.129:33513: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:40:28.944747 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 14584
W20240506 11:40:28.961876 14382 connection.cc:535] server connection from 127.10.45.130:50717 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 11:40:28.961918 14860 connection.cc:535] client connection to 127.10.45.130:46181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 11:40:28.962663 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 15398
W20240506 11:40:28.963236 14860 consensus_peers.cc:485] T f8cb8cbb86424c2d9ec72dc29b001ab0 P d78a1d347bc44a82b3cae400b801bf9d -> Peer 680e178c01b14b439b3fe7e7d2a33038 (127.10.45.130:46181): Couldn't send request to peer 680e178c01b14b439b3fe7e7d2a33038. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.130:46181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:40:28.968727 15530 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:29.062353 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 15398
I20240506 11:40:29.078181 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 14844
I20240506 11:40:29.079072 14965 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:29.166307 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 14844
W20240506 11:40:29.183481 14989 connection.cc:535] server connection from 127.10.45.132:60177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 11:40:29.183931 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 14974
I20240506 11:40:29.184768 15095 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:29.286190 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 14974
I20240506 11:40:29.305610 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 14364
I20240506 11:40:29.306548 14425 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:29.403501 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 14364
2024-05-06T11:40:29Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElection (10865 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica
I20240506 11:40:29.448323 10422 test_util.cc:268] Using random seed: -1991713681
I20240506 11:40:29.449913 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:40:29.449973 10422 ts_itest-base.cc:113] --------------
I20240506 11:40:29.450017 10422 ts_itest-base.cc:114] 1 tablet servers
I20240506 11:40:29.450060 10422 ts_itest-base.cc:115] 1 replicas per TS
I20240506 11:40:29.450114 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:40:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:40:29Z Disabled control of system clock
I20240506 11:40:29.477239 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:35181
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:35181 with env {}
W20240506 11:40:29.770117 15628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:29.770469 15628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:29.770617 15628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:29.777205 15628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:40:29.777294 15628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:29.777348 15628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:40:29.777413 15628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:40:29.785251 15628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:35181
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:35181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:29.786401 15628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:29.788466 15628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:29.801064 15633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:29.801918 15634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:29.802727 15636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:29.803565 15628 server_base.cc:1034] running on GCE node
I20240506 11:40:29.804255 15628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:29.805572 15628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:29.806792 15628 hybrid_clock.cc:648] HybridClock initialized: now 1714995629806728 us; error 36 us; skew 500 ppm
I20240506 11:40:29.807166 15628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:29.809289 15628 webserver.cc:460] Webserver started at http://127.10.45.190:44881/ using document root <none> and password file <none>
I20240506 11:40:29.809955 15628 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:29.810081 15628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:29.810359 15628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:29.812795 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a161337d252a4a07a973bb9c7a59b1a8"
format_stamp: "Formatted at 2024-05-06 11:40:29 on dist-test-slave-7l0g"
I20240506 11:40:29.813452 15628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a161337d252a4a07a973bb9c7a59b1a8"
format_stamp: "Formatted at 2024-05-06 11:40:29 on dist-test-slave-7l0g"
I20240506 11:40:29.818112 15628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240506 11:40:29.820891 15642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:29.822073 15628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240506 11:40:29.822212 15628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a161337d252a4a07a973bb9c7a59b1a8"
format_stamp: "Formatted at 2024-05-06 11:40:29 on dist-test-slave-7l0g"
I20240506 11:40:29.822361 15628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:29.848879 15628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:29.849573 15628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:29.849808 15628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:29.865633 15628 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:35181
I20240506 11:40:29.865679 15693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:35181 every 8 connection(s)
I20240506 11:40:29.866788 15628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:40:29.869954 15694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:29.872542 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 15628
I20240506 11:40:29.872709 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:40:29.879036 15694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8: Bootstrap starting.
I20240506 11:40:29.882319 15694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:29.883409 15694 log.cc:824] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:29.886055 15694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8: No bootstrap required, opened a new log
I20240506 11:40:29.891281 15694 raft_consensus.cc:348] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161337d252a4a07a973bb9c7a59b1a8" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 35181 } }
I20240506 11:40:29.891558 15694 raft_consensus.cc:374] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:29.891651 15694 raft_consensus.cc:729] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a161337d252a4a07a973bb9c7a59b1a8, State: Initialized, Role: FOLLOWER
I20240506 11:40:29.892267 15694 consensus_queue.cc:260] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161337d252a4a07a973bb9c7a59b1a8" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 35181 } }
I20240506 11:40:29.892470 15694 raft_consensus.cc:388] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:40:29.892575 15694 raft_consensus.cc:482] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:40:29.892722 15694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:29.894495 15694 raft_consensus.cc:504] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161337d252a4a07a973bb9c7a59b1a8" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 35181 } }
I20240506 11:40:29.894923 15694 leader_election.cc:302] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a161337d252a4a07a973bb9c7a59b1a8; no voters:
I20240506 11:40:29.895481 15694 leader_election.cc:288] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:40:29.895617 15699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:29.896189 15699 raft_consensus.cc:686] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [term 1 LEADER]: Becoming Leader. State: Replica: a161337d252a4a07a973bb9c7a59b1a8, State: Running, Role: LEADER
I20240506 11:40:29.896689 15699 consensus_queue.cc:237] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161337d252a4a07a973bb9c7a59b1a8" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 35181 } }
I20240506 11:40:29.897385 15694 sys_catalog.cc:564] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:40:29.899269 15701 sys_catalog.cc:455] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a161337d252a4a07a973bb9c7a59b1a8. Latest consensus state: current_term: 1 leader_uuid: "a161337d252a4a07a973bb9c7a59b1a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161337d252a4a07a973bb9c7a59b1a8" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 35181 } } }
I20240506 11:40:29.899264 15700 sys_catalog.cc:455] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a161337d252a4a07a973bb9c7a59b1a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a161337d252a4a07a973bb9c7a59b1a8" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 35181 } } }
I20240506 11:40:29.899629 15700 sys_catalog.cc:458] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:29.899627 15701 sys_catalog.cc:458] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:29.904167 15705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:40:29.909341 15705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:40:29.917078 15705 catalog_manager.cc:1347] Generated new cluster ID: 6c79c0ff8e924b00be5dcec7cb922080
I20240506 11:40:29.917169 15705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:40:29.925125 15705 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:40:29.926223 15705 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:40:29.940196 15705 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a161337d252a4a07a973bb9c7a59b1a8: Generated new TSK 0
I20240506 11:40:29.941087 15705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:40:29.958003 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:35181
--builtin_ntp_servers=127.10.45.148:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20240506 11:40:30.258399 15718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:30.258646 15718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:30.258760 15718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:30.265336 15718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:30.265502 15718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:40:30.273308 15718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:35181
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:30.274413 15718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:30.275954 15718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:30.286980 15723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:30.287875 15726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:30.288780 15724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:30.289614 15718 server_base.cc:1034] running on GCE node
I20240506 11:40:30.290074 15718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:30.290812 15718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:30.292027 15718 hybrid_clock.cc:648] HybridClock initialized: now 1714995630291964 us; error 44 us; skew 500 ppm
I20240506 11:40:30.292284 15718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:30.294234 15718 webserver.cc:460] Webserver started at http://127.10.45.129:38259/ using document root <none> and password file <none>
I20240506 11:40:30.294843 15718 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:30.294981 15718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:30.295249 15718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:30.297516 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1e552a53709c483588be48f11b60515d"
format_stamp: "Formatted at 2024-05-06 11:40:30 on dist-test-slave-7l0g"
I20240506 11:40:30.298192 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1e552a53709c483588be48f11b60515d"
format_stamp: "Formatted at 2024-05-06 11:40:30 on dist-test-slave-7l0g"
I20240506 11:40:30.302544 15718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240506 11:40:30.305480 15732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:30.306655 15718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 11:40:30.306792 15718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1e552a53709c483588be48f11b60515d"
format_stamp: "Formatted at 2024-05-06 11:40:30 on dist-test-slave-7l0g"
I20240506 11:40:30.306931 15718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:30.321475 15718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:30.322127 15718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:30.322384 15718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:30.323202 15718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:30.324434 15718 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:30.324505 15718 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:30.324617 15718 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:30.324668 15718 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:30.352483 15718 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:42571
I20240506 11:40:30.352525 15844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:42571 every 8 connection(s)
I20240506 11:40:30.353642 15718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:40:30.355463 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 15718
I20240506 11:40:30.355639 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:40:30.364743 15845 heartbeater.cc:347] Connected to a master server at 127.10.45.190:35181
I20240506 11:40:30.364986 15845 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:30.365700 15845 heartbeater.cc:511] Master 127.10.45.190:35181 requested a full tablet report, sending...
I20240506 11:40:30.367511 15659 ts_manager.cc:194] Registered new tserver with Master: 1e552a53709c483588be48f11b60515d (127.10.45.129:42571)
I20240506 11:40:30.368917 15659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:37587
I20240506 11:40:30.370636 10422 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20240506 11:40:30.383244 15659 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34870:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240506 11:40:30.405787 15780 tablet_service.cc:1467] Processing CreateTablet for tablet cd748c9e63514e97b318f065fbf2fb42 (DEFAULT_TABLE table=TestTable [id=3cf6e81da1ea4376bd15d70568516768]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:30.406937 15780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd748c9e63514e97b318f065fbf2fb42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:30.416399 15855 tablet_bootstrap.cc:492] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d: Bootstrap starting.
I20240506 11:40:30.418895 15855 tablet_bootstrap.cc:654] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:30.419817 15855 log.cc:824] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:30.421738 15855 tablet_bootstrap.cc:492] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d: No bootstrap required, opened a new log
I20240506 11:40:30.421967 15855 ts_tablet_manager.cc:1367] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240506 11:40:30.427228 15855 raft_consensus.cc:348] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e552a53709c483588be48f11b60515d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 42571 } }
I20240506 11:40:30.427476 15855 raft_consensus.cc:374] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:30.427563 15855 raft_consensus.cc:729] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e552a53709c483588be48f11b60515d, State: Initialized, Role: FOLLOWER
I20240506 11:40:30.428056 15855 consensus_queue.cc:260] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e552a53709c483588be48f11b60515d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 42571 } }
I20240506 11:40:30.428225 15855 raft_consensus.cc:388] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:40:30.428333 15855 raft_consensus.cc:482] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:40:30.428458 15855 raft_consensus.cc:3037] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:30.430104 15855 raft_consensus.cc:504] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e552a53709c483588be48f11b60515d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 42571 } }
I20240506 11:40:30.430469 15855 leader_election.cc:302] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e552a53709c483588be48f11b60515d; no voters:
I20240506 11:40:30.431092 15855 leader_election.cc:288] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:40:30.431224 15857 raft_consensus.cc:2781] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:30.431784 15857 raft_consensus.cc:686] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [term 1 LEADER]: Becoming Leader. State: Replica: 1e552a53709c483588be48f11b60515d, State: Running, Role: LEADER
I20240506 11:40:30.432071 15845 heartbeater.cc:503] Master 127.10.45.190:35181 was elected leader, sending a full tablet report...
I20240506 11:40:30.432323 15857 consensus_queue.cc:237] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e552a53709c483588be48f11b60515d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 42571 } }
I20240506 11:40:30.432457 15855 ts_tablet_manager.cc:1398] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20240506 11:40:30.436779 15659 catalog_manager.cc:5484] T cd748c9e63514e97b318f065fbf2fb42 P 1e552a53709c483588be48f11b60515d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e552a53709c483588be48f11b60515d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "1e552a53709c483588be48f11b60515d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e552a53709c483588be48f11b60515d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 42571 } health_report { overall_health: HEALTHY } } }
I20240506 11:40:30.448125 10422 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20240506 11:40:30.449695 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1e552a53709c483588be48f11b60515d to finish bootstrapping
I20240506 11:40:30.463366 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 15718
I20240506 11:40:30.464244 15840 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:30.540158 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 15718
I20240506 11:40:30.555677 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 15628
I20240506 11:40:30.556675 15689 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:30.647960 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 15628
2024-05-06T11:40:30Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica (1235 ms)
[ RUN ] RaftConsensusElectionITest.LeaderStepDown
I20240506 11:40:30.684230 10422 test_util.cc:268] Using random seed: -1990477790
I20240506 11:40:30.685786 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:40:30.685855 10422 ts_itest-base.cc:113] --------------
I20240506 11:40:30.685920 10422 ts_itest-base.cc:114] 3 tablet servers
I20240506 11:40:30.685977 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:40:30.686033 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:40:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:40:30Z Disabled control of system clock
I20240506 11:40:30.713392 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:45873
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:45873
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20240506 11:40:31.000183 15873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:31.000464 15873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:31.000594 15873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:31.007230 15873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:40:31.007359 15873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:31.007429 15873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:40:31.007506 15873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:40:31.015566 15873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:45873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:45873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:31.016702 15873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:31.018230 15873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:31.028297 15878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:31.029141 15879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:31.030120 15881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:31.031019 15873 server_base.cc:1034] running on GCE node
I20240506 11:40:31.031621 15873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:31.032748 15873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:31.033949 15873 hybrid_clock.cc:648] HybridClock initialized: now 1714995631033901 us; error 34 us; skew 500 ppm
I20240506 11:40:31.034222 15873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:31.036216 15873 webserver.cc:460] Webserver started at http://127.10.45.190:34823/ using document root <none> and password file <none>
I20240506 11:40:31.036825 15873 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:31.036933 15873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:31.037194 15873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:31.039500 15873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86"
format_stamp: "Formatted at 2024-05-06 11:40:31 on dist-test-slave-7l0g"
I20240506 11:40:31.040146 15873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86"
format_stamp: "Formatted at 2024-05-06 11:40:31 on dist-test-slave-7l0g"
I20240506 11:40:31.044435 15873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:40:31.047389 15887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:31.048542 15873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240506 11:40:31.048707 15873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86"
format_stamp: "Formatted at 2024-05-06 11:40:31 on dist-test-slave-7l0g"
I20240506 11:40:31.048849 15873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:31.063982 15873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:31.064713 15873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:31.064961 15873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:31.079941 15873 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:45873
I20240506 11:40:31.079973 15938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:45873 every 8 connection(s)
I20240506 11:40:31.081144 15873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:40:31.084388 15939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:31.089596 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 15873
I20240506 11:40:31.089802 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:40:31.093325 15939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86: Bootstrap starting.
I20240506 11:40:31.095925 15939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:31.096832 15939 log.cc:824] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:31.099027 15939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86: No bootstrap required, opened a new log
I20240506 11:40:31.105077 15939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45873 } }
I20240506 11:40:31.105393 15939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:31.105515 15939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4c4fa5f3d6400aa3ddfabf3f207a86, State: Initialized, Role: FOLLOWER
I20240506 11:40:31.106016 15939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45873 } }
I20240506 11:40:31.106180 15939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:40:31.106283 15939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:40:31.106400 15939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:31.108198 15939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45873 } }
I20240506 11:40:31.108585 15939 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4c4fa5f3d6400aa3ddfabf3f207a86; no voters:
I20240506 11:40:31.109088 15939 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:40:31.109239 15944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:31.109781 15944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4c4fa5f3d6400aa3ddfabf3f207a86, State: Running, Role: LEADER
I20240506 11:40:31.110242 15944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45873 } }
I20240506 11:40:31.110711 15939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:40:31.112928 15945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45873 } } }
I20240506 11:40:31.113425 15945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:31.114514 15946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4c4fa5f3d6400aa3ddfabf3f207a86. Latest consensus state: current_term: 1 leader_uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4c4fa5f3d6400aa3ddfabf3f207a86" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45873 } } }
I20240506 11:40:31.114742 15946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:31.114858 15950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:40:31.119788 15950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:40:31.126596 15950 catalog_manager.cc:1347] Generated new cluster ID: 9d9862568d1542018cb10284e1f5941f
I20240506 11:40:31.126708 15950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:40:31.208477 15950 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:40:31.210151 15950 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:40:31.225677 15950 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3b4c4fa5f3d6400aa3ddfabf3f207a86: Generated new TSK 0
I20240506 11:40:31.226635 15950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:40:31.236877 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45873
--builtin_ntp_servers=127.10.45.148:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20240506 11:40:31.538185 15963 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:31.538506 15963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:31.538597 15963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:31.538710 15963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:31.545732 15963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:31.545938 15963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:40:31.554626 15963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:31.555908 15963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:31.557533 15963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:31.569250 15969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:31.570047 15968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:31.570904 15971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:31.571779 15963 server_base.cc:1034] running on GCE node
I20240506 11:40:31.572300 15963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:31.573112 15963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:31.574302 15963 hybrid_clock.cc:648] HybridClock initialized: now 1714995631574261 us; error 32 us; skew 500 ppm
I20240506 11:40:31.574582 15963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:31.576614 15963 webserver.cc:460] Webserver started at http://127.10.45.129:35617/ using document root <none> and password file <none>
I20240506 11:40:31.577277 15963 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:31.577380 15963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:31.577674 15963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:31.581699 15963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9e8d9e783fcb4ab49d88f814790e3f23"
format_stamp: "Formatted at 2024-05-06 11:40:31 on dist-test-slave-7l0g"
I20240506 11:40:31.582433 15963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9e8d9e783fcb4ab49d88f814790e3f23"
format_stamp: "Formatted at 2024-05-06 11:40:31 on dist-test-slave-7l0g"
I20240506 11:40:31.587505 15963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240506 11:40:31.590433 15977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:31.591770 15963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:31.591902 15963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9e8d9e783fcb4ab49d88f814790e3f23"
format_stamp: "Formatted at 2024-05-06 11:40:31 on dist-test-slave-7l0g"
I20240506 11:40:31.592065 15963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:31.608163 15963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:31.608911 15963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:31.609153 15963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:31.610037 15963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:31.611505 15963 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:31.611594 15963 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:31.611703 15963 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:31.611773 15963 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:31.640672 15963 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:34377
I20240506 11:40:31.640748 16089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:34377 every 8 connection(s)
I20240506 11:40:31.641857 15963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:40:31.647406 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 15963
I20240506 11:40:31.647625 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:40:31.651026 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45873
--builtin_ntp_servers=127.10.45.148:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20240506 11:40:31.654573 16090 heartbeater.cc:347] Connected to a master server at 127.10.45.190:45873
I20240506 11:40:31.654850 16090 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:31.655812 16090 heartbeater.cc:511] Master 127.10.45.190:45873 requested a full tablet report, sending...
I20240506 11:40:31.657886 15904 ts_manager.cc:194] Registered new tserver with Master: 9e8d9e783fcb4ab49d88f814790e3f23 (127.10.45.129:34377)
I20240506 11:40:31.659204 15904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:33633
W20240506 11:40:31.952172 16094 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:31.952481 16094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:31.952564 16094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:31.952677 16094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:31.959473 16094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:31.959637 16094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:40:31.967276 16094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:31.968364 16094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:31.969954 16094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:31.981748 16102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:31.982635 16100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:31.983448 16099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:31.984282 16094 server_base.cc:1034] running on GCE node
I20240506 11:40:31.984889 16094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:31.985868 16094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:31.987074 16094 hybrid_clock.cc:648] HybridClock initialized: now 1714995631987024 us; error 33 us; skew 500 ppm
I20240506 11:40:31.987408 16094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:31.989914 16094 webserver.cc:460] Webserver started at http://127.10.45.130:35929/ using document root <none> and password file <none>
I20240506 11:40:31.990630 16094 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:31.990752 16094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:31.991138 16094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:31.993438 16094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8c5f3e63145b4f0db094d73d1e4f3c01"
format_stamp: "Formatted at 2024-05-06 11:40:31 on dist-test-slave-7l0g"
I20240506 11:40:31.994092 16094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8c5f3e63145b4f0db094d73d1e4f3c01"
format_stamp: "Formatted at 2024-05-06 11:40:31 on dist-test-slave-7l0g"
I20240506 11:40:31.998667 16094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240506 11:40:32.001632 16108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:32.002883 16094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240506 11:40:32.003052 16094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8c5f3e63145b4f0db094d73d1e4f3c01"
format_stamp: "Formatted at 2024-05-06 11:40:31 on dist-test-slave-7l0g"
I20240506 11:40:32.003232 16094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:32.021924 16094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:32.022699 16094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:32.022977 16094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:32.023818 16094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:32.025084 16094 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:32.025162 16094 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:32.025262 16094 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:32.025331 16094 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:32.054384 16094 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:41545
I20240506 11:40:32.054454 16220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:41545 every 8 connection(s)
I20240506 11:40:32.055641 16094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:40:32.060055 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 16094
I20240506 11:40:32.060215 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:40:32.062855 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45873
--builtin_ntp_servers=127.10.45.148:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20240506 11:40:32.067319 16221 heartbeater.cc:347] Connected to a master server at 127.10.45.190:45873
I20240506 11:40:32.067565 16221 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:32.068266 16221 heartbeater.cc:511] Master 127.10.45.190:45873 requested a full tablet report, sending...
I20240506 11:40:32.069672 15904 ts_manager.cc:194] Registered new tserver with Master: 8c5f3e63145b4f0db094d73d1e4f3c01 (127.10.45.130:41545)
I20240506 11:40:32.070689 15904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:37113
W20240506 11:40:32.362064 16224 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:32.362330 16224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:32.362386 16224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:32.362457 16224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:32.368925 16224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:32.369052 16224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:40:32.377131 16224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:32.378286 16224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:32.379807 16224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:32.390698 16229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:32.391563 16232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:32.392369 16230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:32.393301 16224 server_base.cc:1034] running on GCE node
I20240506 11:40:32.393795 16224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:32.394565 16224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:32.395746 16224 hybrid_clock.cc:648] HybridClock initialized: now 1714995632395714 us; error 28 us; skew 500 ppm
I20240506 11:40:32.396085 16224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:32.398070 16224 webserver.cc:460] Webserver started at http://127.10.45.131:40965/ using document root <none> and password file <none>
I20240506 11:40:32.398699 16224 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:32.398797 16224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:32.399111 16224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:32.401321 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0bfe471550f24e44933f337d78118731"
format_stamp: "Formatted at 2024-05-06 11:40:32 on dist-test-slave-7l0g"
I20240506 11:40:32.402050 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0bfe471550f24e44933f337d78118731"
format_stamp: "Formatted at 2024-05-06 11:40:32 on dist-test-slave-7l0g"
I20240506 11:40:32.406523 16224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240506 11:40:32.409531 16238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:32.410661 16224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240506 11:40:32.410789 16224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0bfe471550f24e44933f337d78118731"
format_stamp: "Formatted at 2024-05-06 11:40:32 on dist-test-slave-7l0g"
I20240506 11:40:32.410979 16224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:32.434386 16224 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:32.435176 16224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:32.435420 16224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:32.436259 16224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:32.437542 16224 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:32.437629 16224 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:32.437729 16224 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:32.437793 16224 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:32.465875 16224 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:42113
I20240506 11:40:32.465925 16350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:42113 every 8 connection(s)
I20240506 11:40:32.467132 16224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:40:32.475270 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 16224
I20240506 11:40:32.475402 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:40:32.483172 16351 heartbeater.cc:347] Connected to a master server at 127.10.45.190:45873
I20240506 11:40:32.483450 16351 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:32.484180 16351 heartbeater.cc:511] Master 127.10.45.190:45873 requested a full tablet report, sending...
I20240506 11:40:32.485359 15904 ts_manager.cc:194] Registered new tserver with Master: 0bfe471550f24e44933f337d78118731 (127.10.45.131:42113)
I20240506 11:40:32.486045 15904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:48715
I20240506 11:40:32.489960 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:40:32.502640 15904 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60394:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20240506 11:40:32.507807 15904 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 11:40:32.532296 16286 tablet_service.cc:1467] Processing CreateTablet for tablet 0c498e2a84574334a269dbd5074a38ac (DEFAULT_TABLE table=TestTable [id=8b644cff3c804483b7188df72bfb15b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:32.532656 16156 tablet_service.cc:1467] Processing CreateTablet for tablet 0c498e2a84574334a269dbd5074a38ac (DEFAULT_TABLE table=TestTable [id=8b644cff3c804483b7188df72bfb15b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:32.532656 16025 tablet_service.cc:1467] Processing CreateTablet for tablet 0c498e2a84574334a269dbd5074a38ac (DEFAULT_TABLE table=TestTable [id=8b644cff3c804483b7188df72bfb15b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:32.533463 16286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c498e2a84574334a269dbd5074a38ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:32.533874 16156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c498e2a84574334a269dbd5074a38ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:32.533874 16025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c498e2a84574334a269dbd5074a38ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:32.542622 16369 tablet_bootstrap.cc:492] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731: Bootstrap starting.
I20240506 11:40:32.545578 16369 tablet_bootstrap.cc:654] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:32.545886 16371 tablet_bootstrap.cc:492] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01: Bootstrap starting.
I20240506 11:40:32.547219 16369 log.cc:824] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:32.549266 16371 tablet_bootstrap.cc:654] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:32.550690 16371 log.cc:824] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:32.551609 16370 tablet_bootstrap.cc:492] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23: Bootstrap starting.
I20240506 11:40:32.554288 16371 tablet_bootstrap.cc:492] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01: No bootstrap required, opened a new log
I20240506 11:40:32.554508 16371 ts_tablet_manager.cc:1367] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240506 11:40:32.554543 16369 tablet_bootstrap.cc:492] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731: No bootstrap required, opened a new log
I20240506 11:40:32.554796 16369 ts_tablet_manager.cc:1367] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240506 11:40:32.554850 16370 tablet_bootstrap.cc:654] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:32.556022 16370 log.cc:824] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:32.559332 16370 tablet_bootstrap.cc:492] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23: No bootstrap required, opened a new log
I20240506 11:40:32.559581 16370 ts_tablet_manager.cc:1367] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240506 11:40:32.560791 16371 raft_consensus.cc:348] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41545 } } peers { permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 } } peers { permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 } }
I20240506 11:40:32.561085 16371 raft_consensus.cc:729] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5f3e63145b4f0db094d73d1e4f3c01, State: Initialized, Role: FOLLOWER
I20240506 11:40:32.561493 16371 consensus_queue.cc:260] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41545 } } peers { permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 } } peers { permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 } }
I20240506 11:40:32.562438 16221 heartbeater.cc:503] Master 127.10.45.190:45873 was elected leader, sending a full tablet report...
I20240506 11:40:32.562842 16369 raft_consensus.cc:348] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41545 } } peers { permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 } } peers { permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 } }
I20240506 11:40:32.563254 16369 raft_consensus.cc:729] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bfe471550f24e44933f337d78118731, State: Initialized, Role: FOLLOWER
I20240506 11:40:32.563675 16369 consensus_queue.cc:260] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41545 } } peers { permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 } } peers { permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 } }
I20240506 11:40:32.563827 16371 ts_tablet_manager.cc:1398] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20240506 11:40:32.564719 16351 heartbeater.cc:503] Master 127.10.45.190:45873 was elected leader, sending a full tablet report...
I20240506 11:40:32.565241 16369 ts_tablet_manager.cc:1398] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20240506 11:40:32.568264 16370 raft_consensus.cc:348] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41545 } } peers { permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 } } peers { permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 } }
I20240506 11:40:32.568683 16370 raft_consensus.cc:729] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8d9e783fcb4ab49d88f814790e3f23, State: Initialized, Role: FOLLOWER
I20240506 11:40:32.569123 16370 consensus_queue.cc:260] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41545 } } peers { permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 } } peers { permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 } }
I20240506 11:40:32.570406 16090 heartbeater.cc:503] Master 127.10.45.190:45873 was elected leader, sending a full tablet report...
I20240506 11:40:32.570981 16370 ts_tablet_manager.cc:1398] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20240506 11:40:32.584268 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:40:32.586526 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9e8d9e783fcb4ab49d88f814790e3f23 to finish bootstrapping
I20240506 11:40:32.593966 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8c5f3e63145b4f0db094d73d1e4f3c01 to finish bootstrapping
I20240506 11:40:32.600419 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0bfe471550f24e44933f337d78118731 to finish bootstrapping
I20240506 11:40:32.613260 16176 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0c498e2a84574334a269dbd5074a38ac"
dest_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01"
from {username='slave'} at 127.0.0.1:41822
I20240506 11:40:32.613521 16176 raft_consensus.cc:482] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:40:32.613672 16176 raft_consensus.cc:3037] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:32.615643 16176 raft_consensus.cc:504] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41545 } } peers { permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 } } peers { permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 } }
I20240506 11:40:32.617084 16176 leader_election.cc:288] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [CANDIDATE]: Term 1 election: Requested vote from peers 0bfe471550f24e44933f337d78118731 (127.10.45.131:42113), 9e8d9e783fcb4ab49d88f814790e3f23 (127.10.45.129:34377)
I20240506 11:40:32.623550 16045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c498e2a84574334a269dbd5074a38ac" candidate_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9e8d9e783fcb4ab49d88f814790e3f23"
I20240506 11:40:32.623551 16306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c498e2a84574334a269dbd5074a38ac" candidate_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0bfe471550f24e44933f337d78118731"
I20240506 11:40:32.623939 16045 raft_consensus.cc:3037] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:32.623939 16306 raft_consensus.cc:3037] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:32.626660 16306 raft_consensus.cc:2454] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5f3e63145b4f0db094d73d1e4f3c01 in term 1.
I20240506 11:40:32.626660 16045 raft_consensus.cc:2454] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c5f3e63145b4f0db094d73d1e4f3c01 in term 1.
I20240506 11:40:32.627265 16112 leader_election.cc:302] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c5f3e63145b4f0db094d73d1e4f3c01, 9e8d9e783fcb4ab49d88f814790e3f23; no voters:
I20240506 11:40:32.627630 16375 raft_consensus.cc:2781] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:32.628022 16375 raft_consensus.cc:686] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 1 LEADER]: Becoming Leader. State: Replica: 8c5f3e63145b4f0db094d73d1e4f3c01, State: Running, Role: LEADER
I20240506 11:40:32.628537 16375 consensus_queue.cc:237] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41545 } } peers { permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 } } peers { permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 } }
I20240506 11:40:32.633143 15904 catalog_manager.cc:5484] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c5f3e63145b4f0db094d73d1e4f3c01 (127.10.45.130). New cstate: current_term: 1 leader_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 } health_report { overall_health: UNKNOWN } } }
I20240506 11:40:32.638240 16045 raft_consensus.cc:1261] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23 [term 1 FOLLOWER]: Refusing update from remote peer 8c5f3e63145b4f0db094d73d1e4f3c01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:32.638240 16306 raft_consensus.cc:1261] T 0c498e2a84574334a269dbd5074a38ac P 0bfe471550f24e44933f337d78118731 [term 1 FOLLOWER]: Refusing update from remote peer 8c5f3e63145b4f0db094d73d1e4f3c01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:32.639027 16382 consensus_queue.cc:1035] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:32.639250 16375 consensus_queue.cc:1035] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240506 11:40:32.643863 16091 tablet.cc:2369] T 0c498e2a84574334a269dbd5074a38ac P 9e8d9e783fcb4ab49d88f814790e3f23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:32.663460 16386 mvcc.cc:205] Tried to move back new op lower bound from 7024622111278301184 to 7024622111250984960. Current Snapshot: MvccSnapshot[applied={T|T < 7024622111278301184}]
I20240506 11:40:32.669803 16176 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0c498e2a84574334a269dbd5074a38ac"
dest_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01"
from {username='slave'} at 127.0.0.1:41822
I20240506 11:40:32.670009 16176 raft_consensus.cc:580] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 1 LEADER]: Received request to step down
I20240506 11:40:32.670085 16176 raft_consensus.cc:3032] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:40:32.670168 16176 raft_consensus.cc:729] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c5f3e63145b4f0db094d73d1e4f3c01, State: Running, Role: LEADER
I20240506 11:40:32.670373 16176 consensus_queue.cc:260] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41545 } } peers { permanent_uuid: "0bfe471550f24e44933f337d78118731" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 42113 } } peers { permanent_uuid: "9e8d9e783fcb4ab49d88f814790e3f23" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 34377 } }
I20240506 11:40:32.670766 16176 raft_consensus.cc:3037] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:32.672544 16176 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0c498e2a84574334a269dbd5074a38ac"
dest_uuid: "8c5f3e63145b4f0db094d73d1e4f3c01"
from {username='slave'} at 127.0.0.1:41822
I20240506 11:40:32.672705 16176 raft_consensus.cc:573] T 0c498e2a84574334a269dbd5074a38ac P 8c5f3e63145b4f0db094d73d1e4f3c01 [term 2 FOLLOWER]: Rejecting request to step down while not leader
W20240506 11:40:32.674230 16136 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41822: Illegal state: replica 8c5f3e63145b4f0db094d73d1e4f3c01 is not leader of this config: current role FOLLOWER
I20240506 11:40:32.675348 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 15963
I20240506 11:40:32.676548 16085 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:32.756978 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 15963
I20240506 11:40:32.773030 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 16094
I20240506 11:40:32.774008 16216 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:32.854198 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 16094
I20240506 11:40:32.870218 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 16224
I20240506 11:40:32.871212 16346 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:32.953424 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 16224
I20240506 11:40:32.969647 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 15873
I20240506 11:40:32.970732 15934 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:33.066723 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 15873
2024-05-06T11:40:33Z chronyd exiting
[ OK ] RaftConsensusElectionITest.LeaderStepDown (2419 ms)
[ RUN ] RaftConsensusElectionITest.StepDownWithSlowFollower
I20240506 11:40:33.104231 10422 test_util.cc:268] Using random seed: -1988057775
I20240506 11:40:33.105810 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:40:33.105870 10422 ts_itest-base.cc:113] --------------
I20240506 11:40:33.105921 10422 ts_itest-base.cc:114] 3 tablet servers
I20240506 11:40:33.105963 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:40:33.106019 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:40:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:40:33Z Disabled control of system clock
I20240506 11:40:33.131995 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:44817
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:44817
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20240506 11:40:33.423046 16404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:33.423349 16404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:33.423462 16404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:33.430049 16404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:40:33.430152 16404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:33.430229 16404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:40:33.430301 16404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:40:33.438068 16404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:44817
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:44817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:33.439240 16404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:33.440728 16404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:33.450500 16409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:33.451303 16412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:33.452070 16410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:33.452980 16404 server_base.cc:1034] running on GCE node
I20240506 11:40:33.453642 16404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:33.454833 16404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:33.456075 16404 hybrid_clock.cc:648] HybridClock initialized: now 1714995633455997 us; error 56 us; skew 500 ppm
I20240506 11:40:33.456374 16404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:33.458324 16404 webserver.cc:460] Webserver started at http://127.10.45.190:38639/ using document root <none> and password file <none>
I20240506 11:40:33.458956 16404 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:33.459079 16404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:33.459329 16404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:33.461645 16404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b47f2035c5384a269c22feca29397167"
format_stamp: "Formatted at 2024-05-06 11:40:33 on dist-test-slave-7l0g"
I20240506 11:40:33.462318 16404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b47f2035c5384a269c22feca29397167"
format_stamp: "Formatted at 2024-05-06 11:40:33 on dist-test-slave-7l0g"
I20240506 11:40:33.466743 16404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240506 11:40:33.469552 16418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:33.470762 16404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 11:40:33.470916 16404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b47f2035c5384a269c22feca29397167"
format_stamp: "Formatted at 2024-05-06 11:40:33 on dist-test-slave-7l0g"
I20240506 11:40:33.471107 16404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:33.486909 16404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:33.487615 16404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:33.487856 16404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:33.503899 16404 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:44817
I20240506 11:40:33.503930 16469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:44817 every 8 connection(s)
I20240506 11:40:33.505111 16404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:40:33.506623 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 16404
I20240506 11:40:33.506814 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:40:33.508903 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:33.520747 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167: Bootstrap starting.
I20240506 11:40:33.523391 16470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:33.524230 16470 log.cc:824] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:33.526589 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167: No bootstrap required, opened a new log
I20240506 11:40:33.532101 16470 raft_consensus.cc:348] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2035c5384a269c22feca29397167" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44817 } }
I20240506 11:40:33.532384 16470 raft_consensus.cc:374] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:33.532466 16470 raft_consensus.cc:729] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47f2035c5384a269c22feca29397167, State: Initialized, Role: FOLLOWER
I20240506 11:40:33.532971 16470 consensus_queue.cc:260] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2035c5384a269c22feca29397167" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44817 } }
I20240506 11:40:33.533159 16470 raft_consensus.cc:388] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:40:33.533259 16470 raft_consensus.cc:482] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:40:33.533424 16470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:33.535310 16470 raft_consensus.cc:504] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2035c5384a269c22feca29397167" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44817 } }
I20240506 11:40:33.535691 16470 leader_election.cc:302] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b47f2035c5384a269c22feca29397167; no voters:
I20240506 11:40:33.536187 16470 leader_election.cc:288] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:40:33.536355 16475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:33.536902 16475 raft_consensus.cc:686] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [term 1 LEADER]: Becoming Leader. State: Replica: b47f2035c5384a269c22feca29397167, State: Running, Role: LEADER
I20240506 11:40:33.537380 16475 consensus_queue.cc:237] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2035c5384a269c22feca29397167" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44817 } }
I20240506 11:40:33.537824 16470 sys_catalog.cc:564] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:40:33.539676 16477 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47f2035c5384a269c22feca29397167. Latest consensus state: current_term: 1 leader_uuid: "b47f2035c5384a269c22feca29397167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2035c5384a269c22feca29397167" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44817 } } }
I20240506 11:40:33.539971 16477 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:33.540257 16476 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b47f2035c5384a269c22feca29397167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47f2035c5384a269c22feca29397167" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 44817 } } }
I20240506 11:40:33.540553 16476 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:33.541776 16480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:40:33.547133 16480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:40:33.554620 16480 catalog_manager.cc:1347] Generated new cluster ID: e54aa91db8e34c4da100325172289e79
I20240506 11:40:33.554713 16480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:40:33.598081 16480 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:40:33.599762 16480 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:40:33.610813 16480 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b47f2035c5384a269c22feca29397167: Generated new TSK 0
I20240506 11:40:33.611712 16480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:40:33.626816 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:44817
--builtin_ntp_servers=127.10.45.148:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
W20240506 11:40:33.923612 16494 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:33.923987 16494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:33.924098 16494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:33.924247 16494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:33.934820 16494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:33.934995 16494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:40:33.943441 16494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:44817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:33.944593 16494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:33.946178 16494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:33.957612 16499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:33.958494 16500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:33.959280 16502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:33.960153 16494 server_base.cc:1034] running on GCE node
I20240506 11:40:33.960656 16494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:33.961443 16494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:33.962637 16494 hybrid_clock.cc:648] HybridClock initialized: now 1714995633962586 us; error 45 us; skew 500 ppm
I20240506 11:40:33.962891 16494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:33.964864 16494 webserver.cc:460] Webserver started at http://127.10.45.129:32811/ using document root <none> and password file <none>
I20240506 11:40:33.965515 16494 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:33.965628 16494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:33.965921 16494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:33.968201 16494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "964a6c12cf3a46709a614488eb215113"
format_stamp: "Formatted at 2024-05-06 11:40:33 on dist-test-slave-7l0g"
I20240506 11:40:33.968909 16494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "964a6c12cf3a46709a614488eb215113"
format_stamp: "Formatted at 2024-05-06 11:40:33 on dist-test-slave-7l0g"
I20240506 11:40:33.973369 16494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240506 11:40:33.976418 16508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:33.977536 16494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240506 11:40:33.977672 16494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "964a6c12cf3a46709a614488eb215113"
format_stamp: "Formatted at 2024-05-06 11:40:33 on dist-test-slave-7l0g"
I20240506 11:40:33.977840 16494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:33.992686 16494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:33.993353 16494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:33.993585 16494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:33.994369 16494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:33.995662 16494 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:33.995755 16494 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:33.995860 16494 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:33.995930 16494 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:34.025776 16494 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:37055
I20240506 11:40:34.025838 16620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:37055 every 8 connection(s)
I20240506 11:40:34.026890 16494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:40:34.027328 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 16494
I20240506 11:40:34.027511 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:40:34.031158 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:44817
--builtin_ntp_servers=127.10.45.148:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20240506 11:40:34.038993 16621 heartbeater.cc:347] Connected to a master server at 127.10.45.190:44817
I20240506 11:40:34.039274 16621 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:34.040158 16621 heartbeater.cc:511] Master 127.10.45.190:44817 requested a full tablet report, sending...
I20240506 11:40:34.042258 16435 ts_manager.cc:194] Registered new tserver with Master: 964a6c12cf3a46709a614488eb215113 (127.10.45.129:37055)
I20240506 11:40:34.043715 16435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:42241
W20240506 11:40:34.324534 16625 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:34.324851 16625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:34.324942 16625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:34.325057 16625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:34.331648 16625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:34.331835 16625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:40:34.340001 16625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:44817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:34.341254 16625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:34.342964 16625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:34.354171 16630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:34.355041 16631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:34.355742 16633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:34.356667 16625 server_base.cc:1034] running on GCE node
I20240506 11:40:34.357177 16625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:34.358031 16625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:34.359241 16625 hybrid_clock.cc:648] HybridClock initialized: now 1714995634359198 us; error 29 us; skew 500 ppm
I20240506 11:40:34.359547 16625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:34.361585 16625 webserver.cc:460] Webserver started at http://127.10.45.130:41213/ using document root <none> and password file <none>
I20240506 11:40:34.362262 16625 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:34.362372 16625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:34.362648 16625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:34.365008 16625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "607e7d9c4afd430f92586ac4a1b173d9"
format_stamp: "Formatted at 2024-05-06 11:40:34 on dist-test-slave-7l0g"
I20240506 11:40:34.365741 16625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "607e7d9c4afd430f92586ac4a1b173d9"
format_stamp: "Formatted at 2024-05-06 11:40:34 on dist-test-slave-7l0g"
I20240506 11:40:34.370402 16625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240506 11:40:34.373462 16639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:34.374572 16625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 11:40:34.374716 16625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "607e7d9c4afd430f92586ac4a1b173d9"
format_stamp: "Formatted at 2024-05-06 11:40:34 on dist-test-slave-7l0g"
I20240506 11:40:34.374904 16625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:34.393457 16625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:34.394132 16625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:34.394369 16625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:34.395171 16625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:34.396420 16625 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:34.396497 16625 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:34.396598 16625 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:34.396667 16625 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:34.425513 16625 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:45747
I20240506 11:40:34.425554 16751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:45747 every 8 connection(s)
I20240506 11:40:34.426703 16625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:40:34.430223 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 16625
I20240506 11:40:34.430377 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:40:34.433231 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:44817
--builtin_ntp_servers=127.10.45.148:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20240506 11:40:34.438247 16752 heartbeater.cc:347] Connected to a master server at 127.10.45.190:44817
I20240506 11:40:34.438496 16752 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:34.439363 16752 heartbeater.cc:511] Master 127.10.45.190:44817 requested a full tablet report, sending...
I20240506 11:40:34.440687 16435 ts_manager.cc:194] Registered new tserver with Master: 607e7d9c4afd430f92586ac4a1b173d9 (127.10.45.130:45747)
I20240506 11:40:34.441732 16435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:45463
W20240506 11:40:34.726506 16755 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:34.726794 16755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:34.726871 16755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:34.726966 16755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:34.733482 16755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:34.733611 16755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:40:34.741325 16755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:44817
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:34.742764 16755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:34.744513 16755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:34.755555 16760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:34.756383 16761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:34.757161 16763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:34.758001 16755 server_base.cc:1034] running on GCE node
I20240506 11:40:34.758495 16755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:34.759321 16755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:34.760517 16755 hybrid_clock.cc:648] HybridClock initialized: now 1714995634760474 us; error 41 us; skew 500 ppm
I20240506 11:40:34.760780 16755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:34.762768 16755 webserver.cc:460] Webserver started at http://127.10.45.131:42735/ using document root <none> and password file <none>
I20240506 11:40:34.763433 16755 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:34.763545 16755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:34.763820 16755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:34.766095 16755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "01b9d3bd70ee4de9ab4eb386efa8a913"
format_stamp: "Formatted at 2024-05-06 11:40:34 on dist-test-slave-7l0g"
I20240506 11:40:34.766780 16755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "01b9d3bd70ee4de9ab4eb386efa8a913"
format_stamp: "Formatted at 2024-05-06 11:40:34 on dist-test-slave-7l0g"
I20240506 11:40:34.771150 16755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240506 11:40:34.774096 16769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:34.775267 16755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:34.775398 16755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "01b9d3bd70ee4de9ab4eb386efa8a913"
format_stamp: "Formatted at 2024-05-06 11:40:34 on dist-test-slave-7l0g"
I20240506 11:40:34.775560 16755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:34.797206 16755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:34.797883 16755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:34.798118 16755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:34.798915 16755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:34.800187 16755 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:34.800272 16755 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:34.800371 16755 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:34.800441 16755 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:34.828531 16755 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:41657
I20240506 11:40:34.828581 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:41657 every 8 connection(s)
I20240506 11:40:34.829676 16755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:40:34.832167 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 16755
I20240506 11:40:34.832355 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:40:34.841714 16882 heartbeater.cc:347] Connected to a master server at 127.10.45.190:44817
I20240506 11:40:34.841969 16882 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:34.842754 16882 heartbeater.cc:511] Master 127.10.45.190:44817 requested a full tablet report, sending...
I20240506 11:40:34.844017 16435 ts_manager.cc:194] Registered new tserver with Master: 01b9d3bd70ee4de9ab4eb386efa8a913 (127.10.45.131:41657)
I20240506 11:40:34.844704 16435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:36161
I20240506 11:40:34.847656 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:40:34.860702 16435 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36468:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20240506 11:40:34.865779 16435 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 11:40:34.888993 16817 tablet_service.cc:1467] Processing CreateTablet for tablet e8e4020e88094df1b9efa0514456bba5 (DEFAULT_TABLE table=TestTable [id=78aa2f63d45c4a9e8a458333a8618e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:34.890319 16817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e4020e88094df1b9efa0514456bba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:34.892386 16687 tablet_service.cc:1467] Processing CreateTablet for tablet e8e4020e88094df1b9efa0514456bba5 (DEFAULT_TABLE table=TestTable [id=78aa2f63d45c4a9e8a458333a8618e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:34.892827 16556 tablet_service.cc:1467] Processing CreateTablet for tablet e8e4020e88094df1b9efa0514456bba5 (DEFAULT_TABLE table=TestTable [id=78aa2f63d45c4a9e8a458333a8618e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:34.893653 16687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e4020e88094df1b9efa0514456bba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:34.893960 16556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e4020e88094df1b9efa0514456bba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:34.902329 16900 tablet_bootstrap.cc:492] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113: Bootstrap starting.
I20240506 11:40:34.904193 16901 tablet_bootstrap.cc:492] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913: Bootstrap starting.
I20240506 11:40:34.906505 16900 tablet_bootstrap.cc:654] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:34.907320 16901 tablet_bootstrap.cc:654] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:34.907711 16900 log.cc:824] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:34.908427 16901 log.cc:824] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:34.910352 16900 tablet_bootstrap.cc:492] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113: No bootstrap required, opened a new log
I20240506 11:40:34.910605 16900 ts_tablet_manager.cc:1367] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240506 11:40:34.910861 16901 tablet_bootstrap.cc:492] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913: No bootstrap required, opened a new log
I20240506 11:40:34.911111 16901 ts_tablet_manager.cc:1367] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240506 11:40:34.914178 16902 tablet_bootstrap.cc:492] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9: Bootstrap starting.
I20240506 11:40:34.916630 16902 tablet_bootstrap.cc:654] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:34.917560 16902 log.cc:824] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:34.918882 16900 raft_consensus.cc:348] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 41657 } } peers { permanent_uuid: "607e7d9c4afd430f92586ac4a1b173d9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45747 } } peers { permanent_uuid: "964a6c12cf3a46709a614488eb215113" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37055 } }
I20240506 11:40:34.919306 16900 raft_consensus.cc:729] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964a6c12cf3a46709a614488eb215113, State: Initialized, Role: FOLLOWER
I20240506 11:40:34.919191 16901 raft_consensus.cc:348] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 41657 } } peers { permanent_uuid: "607e7d9c4afd430f92586ac4a1b173d9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45747 } } peers { permanent_uuid: "964a6c12cf3a46709a614488eb215113" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37055 } }
I20240506 11:40:34.919572 16901 raft_consensus.cc:729] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b9d3bd70ee4de9ab4eb386efa8a913, State: Initialized, Role: FOLLOWER
I20240506 11:40:34.919656 16902 tablet_bootstrap.cc:492] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9: No bootstrap required, opened a new log
I20240506 11:40:34.919864 16902 ts_tablet_manager.cc:1367] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240506 11:40:34.919806 16900 consensus_queue.cc:260] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 41657 } } peers { permanent_uuid: "607e7d9c4afd430f92586ac4a1b173d9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45747 } } peers { permanent_uuid: "964a6c12cf3a46709a614488eb215113" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37055 } }
I20240506 11:40:34.920011 16901 consensus_queue.cc:260] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 41657 } } peers { permanent_uuid: "607e7d9c4afd430f92586ac4a1b173d9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45747 } } peers { permanent_uuid: "964a6c12cf3a46709a614488eb215113" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37055 } }
I20240506 11:40:34.920877 16621 heartbeater.cc:503] Master 127.10.45.190:44817 was elected leader, sending a full tablet report...
I20240506 11:40:34.921478 16900 ts_tablet_manager.cc:1398] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240506 11:40:34.922726 16882 heartbeater.cc:503] Master 127.10.45.190:44817 was elected leader, sending a full tablet report...
I20240506 11:40:34.923305 16901 ts_tablet_manager.cc:1398] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20240506 11:40:34.927503 16902 raft_consensus.cc:348] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 41657 } } peers { permanent_uuid: "607e7d9c4afd430f92586ac4a1b173d9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45747 } } peers { permanent_uuid: "964a6c12cf3a46709a614488eb215113" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37055 } }
I20240506 11:40:34.927788 16902 raft_consensus.cc:729] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607e7d9c4afd430f92586ac4a1b173d9, State: Initialized, Role: FOLLOWER
I20240506 11:40:34.928205 16902 consensus_queue.cc:260] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 41657 } } peers { permanent_uuid: "607e7d9c4afd430f92586ac4a1b173d9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45747 } } peers { permanent_uuid: "964a6c12cf3a46709a614488eb215113" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37055 } }
I20240506 11:40:34.929450 16752 heartbeater.cc:503] Master 127.10.45.190:44817 was elected leader, sending a full tablet report...
I20240506 11:40:34.929929 16902 ts_tablet_manager.cc:1398] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240506 11:40:34.941624 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:40:34.943449 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 964a6c12cf3a46709a614488eb215113 to finish bootstrapping
I20240506 11:40:34.951769 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 607e7d9c4afd430f92586ac4a1b173d9 to finish bootstrapping
I20240506 11:40:34.957289 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 01b9d3bd70ee4de9ab4eb386efa8a913 to finish bootstrapping
I20240506 11:40:34.968199 16707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8e4020e88094df1b9efa0514456bba5"
dest_uuid: "607e7d9c4afd430f92586ac4a1b173d9"
from {username='slave'} at 127.0.0.1:41238
I20240506 11:40:34.968473 16707 raft_consensus.cc:482] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:40:34.968632 16707 raft_consensus.cc:3037] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:34.970532 16707 raft_consensus.cc:504] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 41657 } } peers { permanent_uuid: "607e7d9c4afd430f92586ac4a1b173d9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45747 } } peers { permanent_uuid: "964a6c12cf3a46709a614488eb215113" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37055 } }
I20240506 11:40:34.971889 16707 leader_election.cc:288] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [CANDIDATE]: Term 1 election: Requested vote from peers 01b9d3bd70ee4de9ab4eb386efa8a913 (127.10.45.131:41657), 964a6c12cf3a46709a614488eb215113 (127.10.45.129:37055)
I20240506 11:40:34.978713 16837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e4020e88094df1b9efa0514456bba5" candidate_uuid: "607e7d9c4afd430f92586ac4a1b173d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913"
I20240506 11:40:34.979077 16837 raft_consensus.cc:3037] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:34.979977 16576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e4020e88094df1b9efa0514456bba5" candidate_uuid: "607e7d9c4afd430f92586ac4a1b173d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "964a6c12cf3a46709a614488eb215113"
I20240506 11:40:34.980433 16576 raft_consensus.cc:3037] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:34.981384 16837 raft_consensus.cc:2454] T e8e4020e88094df1b9efa0514456bba5 P 01b9d3bd70ee4de9ab4eb386efa8a913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607e7d9c4afd430f92586ac4a1b173d9 in term 1.
I20240506 11:40:34.981974 16641 leader_election.cc:302] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b9d3bd70ee4de9ab4eb386efa8a913, 607e7d9c4afd430f92586ac4a1b173d9; no voters:
I20240506 11:40:34.983114 16908 raft_consensus.cc:2781] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:34.983588 16908 raft_consensus.cc:686] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 1 LEADER]: Becoming Leader. State: Replica: 607e7d9c4afd430f92586ac4a1b173d9, State: Running, Role: LEADER
I20240506 11:40:34.983731 16576 raft_consensus.cc:2454] T e8e4020e88094df1b9efa0514456bba5 P 964a6c12cf3a46709a614488eb215113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607e7d9c4afd430f92586ac4a1b173d9 in term 1.
I20240506 11:40:34.984081 16908 consensus_queue.cc:237] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 41657 } } peers { permanent_uuid: "607e7d9c4afd430f92586ac4a1b173d9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45747 } } peers { permanent_uuid: "964a6c12cf3a46709a614488eb215113" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37055 } }
I20240506 11:40:34.988041 16435 catalog_manager.cc:5484] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 607e7d9c4afd430f92586ac4a1b173d9 (127.10.45.130). New cstate: current_term: 1 leader_uuid: "607e7d9c4afd430f92586ac4a1b173d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 41657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "607e7d9c4afd430f92586ac4a1b173d9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "964a6c12cf3a46709a614488eb215113" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37055 } health_report { overall_health: UNKNOWN } } }
W20240506 11:40:35.178843 16753 tablet.cc:2369] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:36.012152 16707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8e4020e88094df1b9efa0514456bba5"
dest_uuid: "607e7d9c4afd430f92586ac4a1b173d9"
from {username='slave'} at 127.0.0.1:41238
I20240506 11:40:36.012439 16707 raft_consensus.cc:580] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 1 LEADER]: Received request to step down
I20240506 11:40:36.012534 16707 raft_consensus.cc:3032] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:40:36.012629 16707 raft_consensus.cc:729] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 607e7d9c4afd430f92586ac4a1b173d9, State: Running, Role: LEADER
I20240506 11:40:36.012887 16707 consensus_queue.cc:260] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b9d3bd70ee4de9ab4eb386efa8a913" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 41657 } } peers { permanent_uuid: "607e7d9c4afd430f92586ac4a1b173d9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 45747 } } peers { permanent_uuid: "964a6c12cf3a46709a614488eb215113" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 37055 } }
I20240506 11:40:36.013150 16707 raft_consensus.cc:3037] T e8e4020e88094df1b9efa0514456bba5 P 607e7d9c4afd430f92586ac4a1b173d9 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:36.014463 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 16494
W20240506 11:40:36.029915 16641 connection.cc:535] client connection to 127.10.45.129:37055 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 11:40:36.030197 16641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 11:40:36.030511 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 16625
I20240506 11:40:36.031594 16747 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:36.110929 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 16625
I20240506 11:40:36.126701 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 16755
I20240506 11:40:36.141319 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 16404
I20240506 11:40:36.142364 16465 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:36.233573 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 16404
2024-05-06T11:40:36Z chronyd exiting
[ OK ] RaftConsensusElectionITest.StepDownWithSlowFollower (3165 ms)
[ RUN ] RaftConsensusElectionITest.ElectPendingVoter
I20240506 11:40:36.269896 10422 test_util.cc:268] Using random seed: -1984892121
I20240506 11:40:36.271382 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:40:36.271458 10422 ts_itest-base.cc:113] --------------
I20240506 11:40:36.271490 10422 ts_itest-base.cc:114] 5 tablet servers
I20240506 11:40:36.271531 10422 ts_itest-base.cc:115] 5 replicas per TS
I20240506 11:40:36.271564 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:40:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:40:36Z Disabled control of system clock
I20240506 11:40:36.299638 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:46291
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:46291
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20240506 11:40:36.584482 16926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:36.584753 16926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:36.584875 16926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:36.591358 16926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:40:36.591441 16926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:36.591513 16926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:40:36.591583 16926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:40:36.599422 16926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:46291
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:46291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:36.600580 16926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:36.602218 16926 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:36.612607 16932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:36.613536 16931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:36.614231 16934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:36.615145 16926 server_base.cc:1034] running on GCE node
I20240506 11:40:36.615813 16926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:36.617076 16926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:36.618278 16926 hybrid_clock.cc:648] HybridClock initialized: now 1714995636618226 us; error 29 us; skew 500 ppm
I20240506 11:40:36.618577 16926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:36.620774 16926 webserver.cc:460] Webserver started at http://127.10.45.190:46617/ using document root <none> and password file <none>
I20240506 11:40:36.621419 16926 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:36.621542 16926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:36.621835 16926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:36.625609 16926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d41c968bd5cc44bc9d95a339c8172364"
format_stamp: "Formatted at 2024-05-06 11:40:36 on dist-test-slave-7l0g"
I20240506 11:40:36.626322 16926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d41c968bd5cc44bc9d95a339c8172364"
format_stamp: "Formatted at 2024-05-06 11:40:36 on dist-test-slave-7l0g"
I20240506 11:40:36.630832 16926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240506 11:40:36.633738 16940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:36.634907 16926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:36.635133 16926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d41c968bd5cc44bc9d95a339c8172364"
format_stamp: "Formatted at 2024-05-06 11:40:36 on dist-test-slave-7l0g"
I20240506 11:40:36.635293 16926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:36.660071 16926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:36.660802 16926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:36.661116 16926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:36.677423 16926 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:46291
I20240506 11:40:36.677453 16991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:46291 every 8 connection(s)
I20240506 11:40:36.678581 16926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:40:36.681962 16992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:36.684782 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 16926
I20240506 11:40:36.684998 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:40:36.692612 16992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364: Bootstrap starting.
I20240506 11:40:36.695472 16992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:36.696731 16992 log.cc:824] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:36.699156 16992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364: No bootstrap required, opened a new log
I20240506 11:40:36.704530 16992 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c968bd5cc44bc9d95a339c8172364" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 46291 } }
I20240506 11:40:36.704849 16992 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:36.704942 16992 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41c968bd5cc44bc9d95a339c8172364, State: Initialized, Role: FOLLOWER
I20240506 11:40:36.705459 16992 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c968bd5cc44bc9d95a339c8172364" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 46291 } }
I20240506 11:40:36.705626 16992 raft_consensus.cc:388] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:40:36.705747 16992 raft_consensus.cc:482] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:40:36.705905 16992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:36.707943 16992 raft_consensus.cc:504] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c968bd5cc44bc9d95a339c8172364" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 46291 } }
I20240506 11:40:36.708328 16992 leader_election.cc:302] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41c968bd5cc44bc9d95a339c8172364; no voters:
I20240506 11:40:36.708801 16992 leader_election.cc:288] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:40:36.708931 16997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:36.709519 16997 raft_consensus.cc:686] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [term 1 LEADER]: Becoming Leader. State: Replica: d41c968bd5cc44bc9d95a339c8172364, State: Running, Role: LEADER
I20240506 11:40:36.709980 16997 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c968bd5cc44bc9d95a339c8172364" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 46291 } }
I20240506 11:40:36.710500 16992 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:40:36.712464 16998 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41c968bd5cc44bc9d95a339c8172364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c968bd5cc44bc9d95a339c8172364" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 46291 } } }
I20240506 11:40:36.712430 16999 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41c968bd5cc44bc9d95a339c8172364. Latest consensus state: current_term: 1 leader_uuid: "d41c968bd5cc44bc9d95a339c8172364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c968bd5cc44bc9d95a339c8172364" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 46291 } } }
I20240506 11:40:36.712795 16998 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:36.712801 16999 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:36.715130 17004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:40:36.719928 17004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:40:36.727682 17004 catalog_manager.cc:1347] Generated new cluster ID: 4884355c004146838ba7ada7c8717ba7
I20240506 11:40:36.727774 17004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:40:36.733960 17004 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:40:36.735337 17004 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:40:36.759280 17004 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d41c968bd5cc44bc9d95a339c8172364: Generated new TSK 0
I20240506 11:40:36.760288 17004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:40:36.773908 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:46291
--builtin_ntp_servers=127.10.45.148:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20240506 11:40:37.084127 17016 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:37.084453 17016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:37.084546 17016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:37.084659 17016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:37.091483 17016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:37.091732 17016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:40:37.099983 17016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:46291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:37.101225 17016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:37.102872 17016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:37.114173 17021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:37.115077 17022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:37.115931 17024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:37.116739 17016 server_base.cc:1034] running on GCE node
I20240506 11:40:37.117269 17016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:37.118114 17016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:37.119307 17016 hybrid_clock.cc:648] HybridClock initialized: now 1714995637119265 us; error 30 us; skew 500 ppm
I20240506 11:40:37.119567 17016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:37.121579 17016 webserver.cc:460] Webserver started at http://127.10.45.129:41701/ using document root <none> and password file <none>
I20240506 11:40:37.122222 17016 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:37.122308 17016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:37.122545 17016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:37.124754 17016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3d53a39b79af464ba3bb0a49537cd3ca"
format_stamp: "Formatted at 2024-05-06 11:40:37 on dist-test-slave-7l0g"
I20240506 11:40:37.125402 17016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3d53a39b79af464ba3bb0a49537cd3ca"
format_stamp: "Formatted at 2024-05-06 11:40:37 on dist-test-slave-7l0g"
I20240506 11:40:37.130048 17016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240506 11:40:37.132946 17030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:37.134080 17016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 11:40:37.134197 17016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3d53a39b79af464ba3bb0a49537cd3ca"
format_stamp: "Formatted at 2024-05-06 11:40:37 on dist-test-slave-7l0g"
I20240506 11:40:37.134341 17016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:37.148396 17016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:37.149057 17016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:37.149284 17016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:37.150048 17016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:37.151252 17016 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:37.151329 17016 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:37.151398 17016 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:37.151443 17016 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:37.179543 17016 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:32849
I20240506 11:40:37.179613 17142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:32849 every 8 connection(s)
I20240506 11:40:37.180783 17016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:40:37.182818 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 17016
I20240506 11:40:37.183008 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:40:37.186344 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:46291
--builtin_ntp_servers=127.10.45.148:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20240506 11:40:37.193996 17143 heartbeater.cc:347] Connected to a master server at 127.10.45.190:46291
I20240506 11:40:37.194262 17143 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:37.195041 17143 heartbeater.cc:511] Master 127.10.45.190:46291 requested a full tablet report, sending...
I20240506 11:40:37.197196 16957 ts_manager.cc:194] Registered new tserver with Master: 3d53a39b79af464ba3bb0a49537cd3ca (127.10.45.129:32849)
I20240506 11:40:37.199353 16957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:34313
W20240506 11:40:37.491809 17146 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:37.492084 17146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:37.492136 17146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:37.492208 17146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:37.498661 17146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:37.498792 17146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:40:37.506994 17146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:46291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:37.508088 17146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:37.509585 17146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:37.520431 17151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:37.521382 17152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:37.522205 17154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:37.523037 17146 server_base.cc:1034] running on GCE node
I20240506 11:40:37.523566 17146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:37.524340 17146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:37.525532 17146 hybrid_clock.cc:648] HybridClock initialized: now 1714995637525481 us; error 39 us; skew 500 ppm
I20240506 11:40:37.525799 17146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:37.533140 17146 webserver.cc:460] Webserver started at http://127.10.45.130:39765/ using document root <none> and password file <none>
I20240506 11:40:37.533810 17146 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:37.533917 17146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:37.534206 17146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:37.536438 17146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5c0e61e29e474a78a8146a4073fbe582"
format_stamp: "Formatted at 2024-05-06 11:40:37 on dist-test-slave-7l0g"
I20240506 11:40:37.537110 17146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5c0e61e29e474a78a8146a4073fbe582"
format_stamp: "Formatted at 2024-05-06 11:40:37 on dist-test-slave-7l0g"
I20240506 11:40:37.541626 17146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240506 11:40:37.544544 17160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:37.545677 17146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240506 11:40:37.545814 17146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5c0e61e29e474a78a8146a4073fbe582"
format_stamp: "Formatted at 2024-05-06 11:40:37 on dist-test-slave-7l0g"
I20240506 11:40:37.545974 17146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:37.556879 17146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:37.557585 17146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:37.557909 17146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:37.558737 17146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:37.560022 17146 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:37.560101 17146 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:37.560173 17146 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:37.560221 17146 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:37.590121 17146 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:43339
I20240506 11:40:37.590164 17272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:43339 every 8 connection(s)
I20240506 11:40:37.591346 17146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:40:37.598454 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 17146
I20240506 11:40:37.598625 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:40:37.601670 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:46291
--builtin_ntp_servers=127.10.45.148:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20240506 11:40:37.602711 17273 heartbeater.cc:347] Connected to a master server at 127.10.45.190:46291
I20240506 11:40:37.603029 17273 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:37.603909 17273 heartbeater.cc:511] Master 127.10.45.190:46291 requested a full tablet report, sending...
I20240506 11:40:37.605196 16957 ts_manager.cc:194] Registered new tserver with Master: 5c0e61e29e474a78a8146a4073fbe582 (127.10.45.130:43339)
I20240506 11:40:37.605859 16957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:52613
W20240506 11:40:37.892992 17276 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:37.893260 17276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:37.893323 17276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:37.893395 17276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:37.899999 17276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:37.900141 17276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:40:37.908282 17276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:46291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:37.909400 17276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:37.910964 17276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:37.923274 17281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:37.924249 17284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:37.925211 17282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:37.926134 17276 server_base.cc:1034] running on GCE node
I20240506 11:40:37.926676 17276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:37.927574 17276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:37.928758 17276 hybrid_clock.cc:648] HybridClock initialized: now 1714995637928716 us; error 38 us; skew 500 ppm
I20240506 11:40:37.929067 17276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:37.931272 17276 webserver.cc:460] Webserver started at http://127.10.45.131:41211/ using document root <none> and password file <none>
I20240506 11:40:37.931937 17276 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:37.932046 17276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:37.932343 17276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:37.934701 17276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7e5d223cf07640268b9a90258c3693bd"
format_stamp: "Formatted at 2024-05-06 11:40:37 on dist-test-slave-7l0g"
I20240506 11:40:37.935420 17276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7e5d223cf07640268b9a90258c3693bd"
format_stamp: "Formatted at 2024-05-06 11:40:37 on dist-test-slave-7l0g"
I20240506 11:40:37.940215 17276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240506 11:40:37.943233 17290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:37.944391 17276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:37.944530 17276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7e5d223cf07640268b9a90258c3693bd"
format_stamp: "Formatted at 2024-05-06 11:40:37 on dist-test-slave-7l0g"
I20240506 11:40:37.944705 17276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:37.961251 17276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:37.961977 17276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:37.962217 17276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:37.963081 17276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:37.964445 17276 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:37.964529 17276 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:37.964646 17276 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:37.964732 17276 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:37.998590 17276 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:40049
I20240506 11:40:37.998641 17402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:40049 every 8 connection(s)
I20240506 11:40:37.999866 17276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:40:38.001406 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 17276
I20240506 11:40:38.001579 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:40:38.004798 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.132:0
--local_ip_for_outbound_sockets=127.10.45.132
--webserver_interface=127.10.45.132
--webserver_port=0
--tserver_master_addrs=127.10.45.190:46291
--builtin_ntp_servers=127.10.45.148:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20240506 11:40:38.012487 17403 heartbeater.cc:347] Connected to a master server at 127.10.45.190:46291
I20240506 11:40:38.012755 17403 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:38.013543 17403 heartbeater.cc:511] Master 127.10.45.190:46291 requested a full tablet report, sending...
I20240506 11:40:38.014881 16957 ts_manager.cc:194] Registered new tserver with Master: 7e5d223cf07640268b9a90258c3693bd (127.10.45.131:40049)
I20240506 11:40:38.015640 16957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:54901
I20240506 11:40:38.202064 17143 heartbeater.cc:503] Master 127.10.45.190:46291 was elected leader, sending a full tablet report...
W20240506 11:40:38.309528 17406 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:38.309829 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:38.309922 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:38.310048 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:38.317104 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:38.317281 17406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.132
I20240506 11:40:38.325438 17406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.45.132
--webserver_port=0
--tserver_master_addrs=127.10.45.190:46291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.132
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:38.326659 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:38.328291 17406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:38.340140 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:38.341064 17411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:38.341831 17412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:38.342820 17406 server_base.cc:1034] running on GCE node
I20240506 11:40:38.343394 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:38.344235 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:38.345453 17406 hybrid_clock.cc:648] HybridClock initialized: now 1714995638345416 us; error 43 us; skew 500 ppm
I20240506 11:40:38.345724 17406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:38.347790 17406 webserver.cc:460] Webserver started at http://127.10.45.132:37155/ using document root <none> and password file <none>
I20240506 11:40:38.348472 17406 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:38.348560 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:38.348834 17406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:38.351307 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4601bbf40cd34a4f86fad15d22feb07b"
format_stamp: "Formatted at 2024-05-06 11:40:38 on dist-test-slave-7l0g"
I20240506 11:40:38.351977 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4601bbf40cd34a4f86fad15d22feb07b"
format_stamp: "Formatted at 2024-05-06 11:40:38 on dist-test-slave-7l0g"
I20240506 11:40:38.356541 17406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240506 11:40:38.359603 17420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:38.360829 17406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:38.360996 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4601bbf40cd34a4f86fad15d22feb07b"
format_stamp: "Formatted at 2024-05-06 11:40:38 on dist-test-slave-7l0g"
I20240506 11:40:38.361171 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:38.403638 17406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:38.404502 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:38.404742 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:38.405611 17406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:38.406914 17406 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:38.407047 17406 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:38.407150 17406 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:38.407220 17406 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:38.436513 17406 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.132:44717
I20240506 11:40:38.436591 17532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.132:44717 every 8 connection(s)
I20240506 11:40:38.437762 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240506 11:40:38.445155 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 17406
I20240506 11:40:38.445331 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240506 11:40:38.448545 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.133:0
--local_ip_for_outbound_sockets=127.10.45.133
--webserver_interface=127.10.45.133
--webserver_port=0
--tserver_master_addrs=127.10.45.190:46291
--builtin_ntp_servers=127.10.45.148:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20240506 11:40:38.449322 17533 heartbeater.cc:347] Connected to a master server at 127.10.45.190:46291
I20240506 11:40:38.449599 17533 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:38.450310 17533 heartbeater.cc:511] Master 127.10.45.190:46291 requested a full tablet report, sending...
I20240506 11:40:38.451572 16957 ts_manager.cc:194] Registered new tserver with Master: 4601bbf40cd34a4f86fad15d22feb07b (127.10.45.132:44717)
I20240506 11:40:38.452263 16957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.132:41033
I20240506 11:40:38.608381 17273 heartbeater.cc:503] Master 127.10.45.190:46291 was elected leader, sending a full tablet report...
W20240506 11:40:38.743080 17536 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:38.743346 17536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:38.743434 17536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:38.743538 17536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:38.750088 17536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:38.750236 17536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.133
I20240506 11:40:38.757617 17536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.10.45.133
--webserver_port=0
--tserver_master_addrs=127.10.45.190:46291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.133
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:38.758728 17536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:38.760282 17536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:38.771389 17541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:38.772336 17544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:38.773232 17542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:38.773983 17536 server_base.cc:1034] running on GCE node
I20240506 11:40:38.774565 17536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:38.775451 17536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:38.776669 17536 hybrid_clock.cc:648] HybridClock initialized: now 1714995638776626 us; error 39 us; skew 500 ppm
I20240506 11:40:38.776957 17536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:38.779031 17536 webserver.cc:460] Webserver started at http://127.10.45.133:34221/ using document root <none> and password file <none>
I20240506 11:40:38.779707 17536 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:38.779814 17536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:38.780100 17536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:38.782508 17536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "212a9897db2d4642b76cbfb783356072"
format_stamp: "Formatted at 2024-05-06 11:40:38 on dist-test-slave-7l0g"
I20240506 11:40:38.783267 17536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "212a9897db2d4642b76cbfb783356072"
format_stamp: "Formatted at 2024-05-06 11:40:38 on dist-test-slave-7l0g"
I20240506 11:40:38.787794 17536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:40:38.790620 17550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:38.791816 17536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240506 11:40:38.791985 17536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "212a9897db2d4642b76cbfb783356072"
format_stamp: "Formatted at 2024-05-06 11:40:38 on dist-test-slave-7l0g"
I20240506 11:40:38.792176 17536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:38.813338 17536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:38.814016 17536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:38.814244 17536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:38.815053 17536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:38.816305 17536 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:38.816395 17536 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:38.816515 17536 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:38.816598 17536 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:38.845479 17536 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.133:37383
I20240506 11:40:38.845537 17662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.133:37383 every 8 connection(s)
I20240506 11:40:38.846650 17536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240506 11:40:38.856655 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 17536
I20240506 11:40:38.856828 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240506 11:40:38.857525 17663 heartbeater.cc:347] Connected to a master server at 127.10.45.190:46291
I20240506 11:40:38.857762 17663 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:38.858433 17663 heartbeater.cc:511] Master 127.10.45.190:46291 requested a full tablet report, sending...
I20240506 11:40:38.859773 16956 ts_manager.cc:194] Registered new tserver with Master: 212a9897db2d4642b76cbfb783356072 (127.10.45.133:37383)
I20240506 11:40:38.860543 16956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.133:45525
I20240506 11:40:38.871625 10422 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240506 11:40:38.885145 16956 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44086:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20240506 11:40:38.890209 16956 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20240506 11:40:38.919551 17078 tablet_service.cc:1467] Processing CreateTablet for tablet d3c29edd11b14dcfb7c2b40d163ae7a3 (DEFAULT_TABLE table=TestTable [id=ac170792e8be40e5b1a4ebc157c118f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:38.921541 17078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c29edd11b14dcfb7c2b40d163ae7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:38.930547 17338 tablet_service.cc:1467] Processing CreateTablet for tablet d3c29edd11b14dcfb7c2b40d163ae7a3 (DEFAULT_TABLE table=TestTable [id=ac170792e8be40e5b1a4ebc157c118f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:38.930555 17598 tablet_service.cc:1467] Processing CreateTablet for tablet d3c29edd11b14dcfb7c2b40d163ae7a3 (DEFAULT_TABLE table=TestTable [id=ac170792e8be40e5b1a4ebc157c118f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:38.931794 17338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c29edd11b14dcfb7c2b40d163ae7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:38.931840 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c29edd11b14dcfb7c2b40d163ae7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:38.935967 17208 tablet_service.cc:1467] Processing CreateTablet for tablet d3c29edd11b14dcfb7c2b40d163ae7a3 (DEFAULT_TABLE table=TestTable [id=ac170792e8be40e5b1a4ebc157c118f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:38.936844 17468 tablet_service.cc:1467] Processing CreateTablet for tablet d3c29edd11b14dcfb7c2b40d163ae7a3 (DEFAULT_TABLE table=TestTable [id=ac170792e8be40e5b1a4ebc157c118f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:38.937319 17208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c29edd11b14dcfb7c2b40d163ae7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:38.938038 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c29edd11b14dcfb7c2b40d163ae7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:38.951103 17685 tablet_bootstrap.cc:492] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd: Bootstrap starting.
I20240506 11:40:38.954206 17689 tablet_bootstrap.cc:492] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582: Bootstrap starting.
I20240506 11:40:38.955899 17685 tablet_bootstrap.cc:654] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:38.956182 17687 tablet_bootstrap.cc:492] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca: Bootstrap starting.
I20240506 11:40:38.956749 17686 tablet_bootstrap.cc:492] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072: Bootstrap starting.
I20240506 11:40:38.958396 17685 log.cc:824] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:38.959264 17687 tablet_bootstrap.cc:654] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:38.960968 17687 log.cc:824] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:38.961460 17686 tablet_bootstrap.cc:654] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:38.962608 17686 log.cc:824] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:38.963492 17688 tablet_bootstrap.cc:492] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b: Bootstrap starting.
I20240506 11:40:38.963771 17689 tablet_bootstrap.cc:654] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:38.964928 17689 log.cc:824] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:38.967418 17688 tablet_bootstrap.cc:654] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:38.968751 17688 log.cc:824] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:38.970000 17685 tablet_bootstrap.cc:492] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd: No bootstrap required, opened a new log
I20240506 11:40:38.970311 17685 ts_tablet_manager.cc:1367] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20240506 11:40:38.973172 17687 tablet_bootstrap.cc:492] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca: No bootstrap required, opened a new log
I20240506 11:40:38.973500 17687 ts_tablet_manager.cc:1367] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.011s
I20240506 11:40:38.976986 17686 tablet_bootstrap.cc:492] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072: No bootstrap required, opened a new log
I20240506 11:40:38.977236 17686 ts_tablet_manager.cc:1367] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240506 11:40:38.980541 17689 tablet_bootstrap.cc:492] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582: No bootstrap required, opened a new log
I20240506 11:40:38.980947 17689 ts_tablet_manager.cc:1367] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.005s
I20240506 11:40:38.983464 17687 raft_consensus.cc:348] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:38.983784 17688 tablet_bootstrap.cc:492] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b: No bootstrap required, opened a new log
I20240506 11:40:38.983989 17687 raft_consensus.cc:729] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d53a39b79af464ba3bb0a49537cd3ca, State: Initialized, Role: FOLLOWER
I20240506 11:40:38.984040 17688 ts_tablet_manager.cc:1367] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.012s
I20240506 11:40:38.984977 17686 raft_consensus.cc:348] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:38.985719 17687 consensus_queue.cc:260] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:38.988791 17689 raft_consensus.cc:348] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:38.989266 17689 raft_consensus.cc:729] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c0e61e29e474a78a8146a4073fbe582, State: Initialized, Role: FOLLOWER
I20240506 11:40:38.989732 17689 consensus_queue.cc:260] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:38.989938 17685 raft_consensus.cc:348] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:38.990506 17686 raft_consensus.cc:729] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212a9897db2d4642b76cbfb783356072, State: Initialized, Role: FOLLOWER
I20240506 11:40:38.991096 17686 consensus_queue.cc:260] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:38.991763 17685 raft_consensus.cc:729] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5d223cf07640268b9a90258c3693bd, State: Initialized, Role: FOLLOWER
I20240506 11:40:38.992218 17688 raft_consensus.cc:348] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:38.992372 17685 consensus_queue.cc:260] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:38.992774 17688 raft_consensus.cc:729] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4601bbf40cd34a4f86fad15d22feb07b, State: Initialized, Role: FOLLOWER
I20240506 11:40:38.993363 17688 consensus_queue.cc:260] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:38.993978 17687 ts_tablet_manager.cc:1398] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca: Time spent starting tablet: real 0.020s user 0.009s sys 0.007s
I20240506 11:40:38.997542 17689 ts_tablet_manager.cc:1398] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20240506 11:40:39.000774 17533 heartbeater.cc:503] Master 127.10.45.190:46291 was elected leader, sending a full tablet report...
I20240506 11:40:39.001605 17688 ts_tablet_manager.cc:1398] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20240506 11:40:39.010457 10422 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240506 11:40:39.012518 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3d53a39b79af464ba3bb0a49537cd3ca to finish bootstrapping
I20240506 11:40:39.017791 17403 heartbeater.cc:503] Master 127.10.45.190:46291 was elected leader, sending a full tablet report...
I20240506 11:40:39.021633 17686 ts_tablet_manager.cc:1398] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072: Time spent starting tablet: real 0.044s user 0.013s sys 0.010s
I20240506 11:40:39.023185 17663 heartbeater.cc:503] Master 127.10.45.190:46291 was elected leader, sending a full tablet report...
I20240506 11:40:39.024152 17685 ts_tablet_manager.cc:1398] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd: Time spent starting tablet: real 0.054s user 0.019s sys 0.005s
I20240506 11:40:39.034531 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5c0e61e29e474a78a8146a4073fbe582 to finish bootstrapping
I20240506 11:40:39.041131 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7e5d223cf07640268b9a90258c3693bd to finish bootstrapping
I20240506 11:40:39.047430 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4601bbf40cd34a4f86fad15d22feb07b to finish bootstrapping
I20240506 11:40:39.053751 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 212a9897db2d4642b76cbfb783356072 to finish bootstrapping
I20240506 11:40:39.064952 17228 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3"
dest_uuid: "5c0e61e29e474a78a8146a4073fbe582"
from {username='slave'} at 127.0.0.1:59562
I20240506 11:40:39.065205 17228 raft_consensus.cc:482] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:40:39.065336 17228 raft_consensus.cc:3037] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:39.067363 17228 raft_consensus.cc:504] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:39.068964 17228 leader_election.cc:288] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [CANDIDATE]: Term 1 election: Requested vote from peers 3d53a39b79af464ba3bb0a49537cd3ca (127.10.45.129:32849), 212a9897db2d4642b76cbfb783356072 (127.10.45.133:37383), 4601bbf40cd34a4f86fad15d22feb07b (127.10.45.132:44717), 7e5d223cf07640268b9a90258c3693bd (127.10.45.131:40049)
I20240506 11:40:39.079718 17358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3" candidate_uuid: "5c0e61e29e474a78a8146a4073fbe582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e5d223cf07640268b9a90258c3693bd"
I20240506 11:40:39.080116 17358 raft_consensus.cc:3037] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:39.083181 17358 raft_consensus.cc:2454] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c0e61e29e474a78a8146a4073fbe582 in term 1.
I20240506 11:40:39.083676 17098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3" candidate_uuid: "5c0e61e29e474a78a8146a4073fbe582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d53a39b79af464ba3bb0a49537cd3ca"
I20240506 11:40:39.084110 17098 raft_consensus.cc:3037] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:39.085523 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3" candidate_uuid: "5c0e61e29e474a78a8146a4073fbe582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4601bbf40cd34a4f86fad15d22feb07b"
I20240506 11:40:39.085917 17488 raft_consensus.cc:3037] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:39.086668 17098 raft_consensus.cc:2454] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c0e61e29e474a78a8146a4073fbe582 in term 1.
I20240506 11:40:39.087215 17164 leader_election.cc:302] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3d53a39b79af464ba3bb0a49537cd3ca, 5c0e61e29e474a78a8146a4073fbe582, 7e5d223cf07640268b9a90258c3693bd; no voters:
I20240506 11:40:39.087401 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3" candidate_uuid: "5c0e61e29e474a78a8146a4073fbe582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "212a9897db2d4642b76cbfb783356072"
I20240506 11:40:39.087603 17696 raft_consensus.cc:2781] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:39.087796 17618 raft_consensus.cc:3037] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:39.088132 17696 raft_consensus.cc:686] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 1 LEADER]: Becoming Leader. State: Replica: 5c0e61e29e474a78a8146a4073fbe582, State: Running, Role: LEADER
I20240506 11:40:39.088904 17488 raft_consensus.cc:2454] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c0e61e29e474a78a8146a4073fbe582 in term 1.
I20240506 11:40:39.088725 17696 consensus_queue.cc:237] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }
I20240506 11:40:39.090385 17618 raft_consensus.cc:2454] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c0e61e29e474a78a8146a4073fbe582 in term 1.
W20240506 11:40:39.093642 17274 tablet.cc:2369] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:39.093765 16957 catalog_manager.cc:5484] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c0e61e29e474a78a8146a4073fbe582 (127.10.45.130). New cstate: current_term: 1 leader_uuid: "5c0e61e29e474a78a8146a4073fbe582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } health_report { overall_health: UNKNOWN } } }
W20240506 11:40:39.098062 17664 tablet.cc:2369] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 11:40:39.183491 17144 tablet.cc:2369] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 11:40:39.189468 17534 tablet.cc:2369] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 11:40:39.252251 17404 tablet.cc:2369] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:39.584092 17696 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:39.627130 17696 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:39.634682 17714 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:39.635675 17696 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:39.665659 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 16926
I20240506 11:40:39.667320 16987 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:39.765784 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 16926
I20240506 11:40:39.783074 10422 raft_consensus_election-itest.cc:666] Removing tserver with uuid 7e5d223cf07640268b9a90258c3693bd
W20240506 11:40:39.783046 17143 heartbeater.cc:644] Failed to heartbeat to 127.10.45.190:46291 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.10.45.190:46291 (error 108)
I20240506 11:40:39.785101 17228 consensus_queue.cc:237] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } }
I20240506 11:40:39.787083 17488 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 1 FOLLOWER]: Refusing update from remote peer 5c0e61e29e474a78a8146a4073fbe582: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:39.787094 17098 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 1 FOLLOWER]: Refusing update from remote peer 5c0e61e29e474a78a8146a4073fbe582: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:39.787197 17618 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 1 FOLLOWER]: Refusing update from remote peer 5c0e61e29e474a78a8146a4073fbe582: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:39.787606 17712 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 11:40:39.787843 17716 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 11:40:39.788092 17714 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 11:40:39.790315 17714 raft_consensus.cc:2932] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7e5d223cf07640268b9a90258c3693bd (127.10.45.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } }
I20240506 11:40:39.790905 17098 raft_consensus.cc:2932] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7e5d223cf07640268b9a90258c3693bd (127.10.45.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } }
I20240506 11:40:39.791173 17488 raft_consensus.cc:2932] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7e5d223cf07640268b9a90258c3693bd (127.10.45.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } }
I20240506 11:40:39.795679 17618 raft_consensus.cc:2932] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7e5d223cf07640268b9a90258c3693bd (127.10.45.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } }
W20240506 11:40:39.800411 17533 heartbeater.cc:644] Failed to heartbeat to 127.10.45.190:46291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.45.190:46291: connect: Connection refused (error 111)
W20240506 11:40:39.801069 17663 heartbeater.cc:644] Failed to heartbeat to 127.10.45.190:46291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.45.190:46291: connect: Connection refused (error 111)
W20240506 11:40:39.807842 17273 heartbeater.cc:644] Failed to heartbeat to 127.10.45.190:46291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.45.190:46291: connect: Connection refused (error 111)
I20240506 11:40:39.823068 10422 raft_consensus_election-itest.cc:676] Pausing 3 replicas...
I20240506 11:40:39.855686 10422 raft_consensus_election-itest.cc:684] Adding back Peer 7e5d223cf07640268b9a90258c3693bd and expecting timeout...
I20240506 11:40:39.857414 17228 consensus_queue.cc:237] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } }
I20240506 11:40:39.859371 17358 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 1 FOLLOWER]: Refusing update from remote peer 5c0e61e29e474a78a8146a4073fbe582: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 11:40:39.859865 17696 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240506 11:40:39.861964 17358 raft_consensus.cc:858] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } }
W20240506 11:40:39.862077 17358 raft_consensus.cc:1565] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 3' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } }
I20240506 11:40:39.862610 17358 raft_consensus.cc:2932] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7e5d223cf07640268b9a90258c3693bd (127.10.45.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } }
I20240506 11:40:39.866443 17358 raft_consensus.cc:1467] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
W20240506 11:40:39.867086 17403 heartbeater.cc:644] Failed to heartbeat to 127.10.45.190:46291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.45.190:46291: connect: Connection refused (error 111)
I20240506 11:40:39.957168 10422 raft_consensus_election-itest.cc:688] Timeout achieved.
I20240506 11:40:39.964428 10422 raft_consensus_election-itest.cc:699] Forcing Peer 5c0e61e29e474a78a8146a4073fbe582 to step down...
I20240506 11:40:39.965034 17228 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3"
dest_uuid: "5c0e61e29e474a78a8146a4073fbe582"
from {username='slave'} at 127.0.0.1:59562
I20240506 11:40:39.965246 17228 raft_consensus.cc:580] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 1 LEADER]: Received request to step down
I20240506 11:40:39.965320 17228 raft_consensus.cc:3032] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:40:39.965385 17228 raft_consensus.cc:729] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5c0e61e29e474a78a8146a4073fbe582, State: Running, Role: LEADER
I20240506 11:40:39.965616 17228 consensus_queue.cc:260] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } }
I20240506 11:40:39.965912 17228 raft_consensus.cc:3037] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:39.966398 10422 raft_consensus_election-itest.cc:705] Resuming Peer 3d53a39b79af464ba3bb0a49537cd3ca ...
I20240506 11:40:39.967059 10422 raft_consensus_election-itest.cc:710] Trying to elect Peer 7e5d223cf07640268b9a90258c3693bd ...
I20240506 11:40:39.967555 17098 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 1 FOLLOWER]: Refusing update from remote peer 5c0e61e29e474a78a8146a4073fbe582: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 11:40:39.967897 17358 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3"
dest_uuid: "7e5d223cf07640268b9a90258c3693bd"
from {username='slave'} at 127.0.0.1:36830
I20240506 11:40:39.968174 17358 raft_consensus.cc:482] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:40:39.968309 17358 raft_consensus.cc:3037] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:39.970201 17358 raft_consensus.cc:504] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } }
I20240506 11:40:39.974638 17358 leader_election.cc:288] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [CANDIDATE]: Term 2 election: Requested vote from peers 3d53a39b79af464ba3bb0a49537cd3ca (127.10.45.129:32849), 212a9897db2d4642b76cbfb783356072 (127.10.45.133:37383), 5c0e61e29e474a78a8146a4073fbe582 (127.10.45.130:43339), 4601bbf40cd34a4f86fad15d22feb07b (127.10.45.132:44717)
I20240506 11:40:39.981660 17098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3" candidate_uuid: "7e5d223cf07640268b9a90258c3693bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3d53a39b79af464ba3bb0a49537cd3ca"
I20240506 11:40:39.981876 17098 raft_consensus.cc:1799] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 1 FOLLOWER]: Handling vote request from an unknown peer 7e5d223cf07640268b9a90258c3693bd
I20240506 11:40:39.982002 17098 raft_consensus.cc:3037] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:39.984845 17098 raft_consensus.cc:2454] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5d223cf07640268b9a90258c3693bd in term 2.
I20240506 11:40:39.990650 10422 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 1.3
I20240506 11:40:39.992653 17228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3" candidate_uuid: "7e5d223cf07640268b9a90258c3693bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5c0e61e29e474a78a8146a4073fbe582"
I20240506 11:40:39.994558 17228 raft_consensus.cc:2454] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5d223cf07640268b9a90258c3693bd in term 2.
I20240506 11:40:39.994978 17293 leader_election.cc:302] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3d53a39b79af464ba3bb0a49537cd3ca, 5c0e61e29e474a78a8146a4073fbe582, 7e5d223cf07640268b9a90258c3693bd; no voters:
I20240506 11:40:39.995411 17708 raft_consensus.cc:2781] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 2 FOLLOWER]: Leader election won for term 2
I20240506 11:40:39.995863 17708 raft_consensus.cc:686] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 2 LEADER]: Becoming Leader. State: Replica: 7e5d223cf07640268b9a90258c3693bd, State: Running, Role: LEADER
I20240506 11:40:39.996384 17708 consensus_queue.cc:237] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } }
I20240506 11:40:40.093765 10422 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20240506 11:40:40.296773 10422 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20240506 11:40:40.437288 17098 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 2 FOLLOWER]: Refusing update from remote peer 7e5d223cf07640268b9a90258c3693bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240506 11:40:40.437940 17708 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:40:40.509339 17228 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 2 FOLLOWER]: Refusing update from remote peer 7e5d223cf07640268b9a90258c3693bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240506 11:40:40.510035 17708 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:40:40.515285 17743 raft_consensus.cc:2932] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [term 2 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7e5d223cf07640268b9a90258c3693bd (127.10.45.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } } }
I20240506 11:40:40.515755 17228 raft_consensus.cc:2932] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 5c0e61e29e474a78a8146a4073fbe582 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7e5d223cf07640268b9a90258c3693bd (127.10.45.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } } }
W20240506 11:40:40.518126 17228 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59562 (request call id 56) took 661 ms (client timeout 100 ms). Trace:
W20240506 11:40:40.518311 17228 rpcz_store.cc:269] 0506 11:40:39.856576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:39.856671 (+ 95us) service_pool.cc:225] Handling call
0506 11:40:39.858317 (+ 1646us) log.cc:842] Serialized 609 byte log entry
0506 11:40:40.518072 (+659755us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48,"raft.queue_time_us":558,"raft.run_cpu_time_us":628,"raft.run_wall_time_us":777,"spinlock_wait_cycles":76160}
I20240506 11:40:40.519492 17098 raft_consensus.cc:2932] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 3d53a39b79af464ba3bb0a49537cd3ca [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7e5d223cf07640268b9a90258c3693bd (127.10.45.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } } }
I20240506 11:40:40.599907 10422 raft_consensus_election-itest.cc:716] Resuming remaining nodes...
I20240506 11:40:40.602582 17618 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 1 FOLLOWER]: Refusing update from remote peer 5c0e61e29e474a78a8146a4073fbe582: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 11:40:40.603600 17488 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 1 FOLLOWER]: Refusing update from remote peer 5c0e61e29e474a78a8146a4073fbe582: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240506 11:40:40.610762 17487 raft_consensus.cc:3037] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:40.613788 17487 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 2 FOLLOWER]: Refusing update from remote peer 7e5d223cf07640268b9a90258c3693bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240506 11:40:40.614213 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3" candidate_uuid: "7e5d223cf07640268b9a90258c3693bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4601bbf40cd34a4f86fad15d22feb07b"
I20240506 11:40:40.614413 17488 raft_consensus.cc:1799] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 2 FOLLOWER]: Handling vote request from an unknown peer 7e5d223cf07640268b9a90258c3693bd
I20240506 11:40:40.617197 17488 raft_consensus.cc:2454] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5d223cf07640268b9a90258c3693bd in term 2.
I20240506 11:40:40.617729 17747 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:40:40.627980 17618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c29edd11b14dcfb7c2b40d163ae7a3" candidate_uuid: "7e5d223cf07640268b9a90258c3693bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "212a9897db2d4642b76cbfb783356072"
I20240506 11:40:40.628168 17618 raft_consensus.cc:1799] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 1 FOLLOWER]: Handling vote request from an unknown peer 7e5d223cf07640268b9a90258c3693bd
I20240506 11:40:40.628283 17618 raft_consensus.cc:3037] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:40.630893 17618 raft_consensus.cc:2454] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5d223cf07640268b9a90258c3693bd in term 2.
I20240506 11:40:40.631712 17618 raft_consensus.cc:1261] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 2 FOLLOWER]: Refusing update from remote peer 7e5d223cf07640268b9a90258c3693bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240506 11:40:40.632221 17708 consensus_queue.cc:1035] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 7e5d223cf07640268b9a90258c3693bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240506 11:40:40.649075 17488 raft_consensus.cc:2932] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 4601bbf40cd34a4f86fad15d22feb07b [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7e5d223cf07640268b9a90258c3693bd (127.10.45.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } } }
I20240506 11:40:40.651698 17618 raft_consensus.cc:2932] T d3c29edd11b14dcfb7c2b40d163ae7a3 P 212a9897db2d4642b76cbfb783356072 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7e5d223cf07640268b9a90258c3693bd (127.10.45.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d53a39b79af464ba3bb0a49537cd3ca" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 32849 } } peers { permanent_uuid: "212a9897db2d4642b76cbfb783356072" member_type: VOTER last_known_addr { host: "127.10.45.133" port: 37383 } } peers { permanent_uuid: "5c0e61e29e474a78a8146a4073fbe582" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 43339 } } peers { permanent_uuid: "4601bbf40cd34a4f86fad15d22feb07b" member_type: VOTER last_known_addr { host: "127.10.45.132" port: 44717 } } peers { permanent_uuid: "7e5d223cf07640268b9a90258c3693bd" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 40049 } attrs { } } }
I20240506 11:40:40.660425 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 17016
I20240506 11:40:40.661357 17138 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:40.741220 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 17016
I20240506 11:40:40.757675 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 17146
I20240506 11:40:40.758679 17268 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:40.840665 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 17146
I20240506 11:40:40.857894 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 17276
I20240506 11:40:40.858878 17398 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:40.943017 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 17276
I20240506 11:40:40.959581 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 17406
I20240506 11:40:40.960644 17528 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:41.040158 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 17406
I20240506 11:40:41.055557 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 17536
I20240506 11:40:41.056540 17658 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:41.136255 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 17536
2024-05-06T11:40:41Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ElectPendingVoter (4902 ms)
[ RUN ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy
I20240506 11:40:41.172508 10422 test_util.cc:268] Using random seed: -1979989511
I20240506 11:40:41.174240 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:40:41.174310 10422 ts_itest-base.cc:113] --------------
I20240506 11:40:41.174357 10422 ts_itest-base.cc:114] 3 tablet servers
I20240506 11:40:41.174413 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:40:41.174458 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:40:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:40:41Z Disabled control of system clock
I20240506 11:40:41.203294 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:38885
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:38885
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20240506 11:40:41.502954 17775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:41.503225 17775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:41.503336 17775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:41.510079 17775 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240506 11:40:41.510208 17775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:40:41.510260 17775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:41.510313 17775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:40:41.510375 17775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:40:41.518350 17775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:38885
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:38885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:41.519644 17775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:41.521237 17775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:41.531540 17780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:41.532331 17783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:41.533095 17781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:41.534011 17775 server_base.cc:1034] running on GCE node
I20240506 11:40:41.534657 17775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:41.535935 17775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:41.537114 17775 hybrid_clock.cc:648] HybridClock initialized: now 1714995641537079 us; error 34 us; skew 500 ppm
I20240506 11:40:41.537379 17775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:41.539312 17775 webserver.cc:460] Webserver started at http://127.10.45.190:34349/ using document root <none> and password file <none>
I20240506 11:40:41.539923 17775 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:41.540009 17775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:41.540243 17775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:41.542539 17775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1593b0c3f63d4da38dbb73dc02c60e24"
format_stamp: "Formatted at 2024-05-06 11:40:41 on dist-test-slave-7l0g"
I20240506 11:40:41.543232 17775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1593b0c3f63d4da38dbb73dc02c60e24"
format_stamp: "Formatted at 2024-05-06 11:40:41 on dist-test-slave-7l0g"
I20240506 11:40:41.547642 17775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:40:41.550679 17789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:41.551944 17775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240506 11:40:41.552078 17775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1593b0c3f63d4da38dbb73dc02c60e24"
format_stamp: "Formatted at 2024-05-06 11:40:41 on dist-test-slave-7l0g"
I20240506 11:40:41.552239 17775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:41.578131 17775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:41.578831 17775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:41.579084 17775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:41.595568 17775 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:38885
I20240506 11:40:41.595590 17840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:38885 every 8 connection(s)
I20240506 11:40:41.596805 17775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:40:41.600176 17841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:41.600801 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 17775
I20240506 11:40:41.601032 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:40:41.612577 17841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24: Bootstrap starting.
I20240506 11:40:41.615845 17841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:41.616694 17841 log.cc:824] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:41.618867 17841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24: No bootstrap required, opened a new log
I20240506 11:40:41.624661 17841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } }
I20240506 11:40:41.625030 17841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:41.625139 17841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1593b0c3f63d4da38dbb73dc02c60e24, State: Initialized, Role: FOLLOWER
I20240506 11:40:41.625730 17841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } }
I20240506 11:40:41.625913 17841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:40:41.626006 17841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:40:41.626117 17841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:41.628544 17841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } }
I20240506 11:40:41.629007 17841 leader_election.cc:302] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1593b0c3f63d4da38dbb73dc02c60e24; no voters:
I20240506 11:40:41.629508 17841 leader_election.cc:288] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:40:41.629668 17846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:41.630251 17846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 1 LEADER]: Becoming Leader. State: Replica: 1593b0c3f63d4da38dbb73dc02c60e24, State: Running, Role: LEADER
I20240506 11:40:41.630775 17846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } }
I20240506 11:40:41.631481 17841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:40:41.633545 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } } }
I20240506 11:40:41.634050 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:41.634788 17848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1593b0c3f63d4da38dbb73dc02c60e24. Latest consensus state: current_term: 1 leader_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } } }
I20240506 11:40:41.635008 17848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:41.635531 17856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:40:41.640161 17856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:40:41.647783 17856 catalog_manager.cc:1347] Generated new cluster ID: cf2768c9620346fabd342c9bf862f297
I20240506 11:40:41.647881 17856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:40:41.663105 17856 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:40:41.664419 17856 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:40:41.673743 17856 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24: Generated new TSK 0
I20240506 11:40:41.674721 17856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:40:41.690258 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:38885
--builtin_ntp_servers=127.10.45.148:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20240506 11:40:41.999876 17865 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20240506 11:40:42.000236 17865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:42.000329 17865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:42.000448 17865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:42.007704 17865 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20240506 11:40:42.007783 17865 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20240506 11:40:42.007869 17865 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240506 11:40:42.007943 17865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:42.008109 17865 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20240506 11:40:42.008176 17865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:40:42.016690 17865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:38885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:42.017938 17865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:42.019551 17865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:42.030683 17871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:42.031483 17873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:42.032248 17870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:42.033087 17865 server_base.cc:1034] running on GCE node
I20240506 11:40:42.033628 17865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:42.034431 17865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:42.035615 17865 hybrid_clock.cc:648] HybridClock initialized: now 1714995642035576 us; error 37 us; skew 500 ppm
I20240506 11:40:42.035884 17865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:42.037940 17865 webserver.cc:460] Webserver started at http://127.10.45.129:37269/ using document root <none> and password file <none>
I20240506 11:40:42.038622 17865 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:42.038734 17865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:42.039081 17865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:42.041556 17865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e17b56ac16374923a53f885332de43e5"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:42.042244 17865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e17b56ac16374923a53f885332de43e5"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:42.046828 17865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240506 11:40:42.049779 17879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:42.050916 17865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:42.051095 17865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e17b56ac16374923a53f885332de43e5"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:42.051271 17865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:42.073772 17865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:42.074544 17865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:42.074823 17865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:42.075762 17865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:42.077138 17865 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:42.077243 17865 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:42.077347 17865 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:42.077414 17865 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:42.106498 17865 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:33737
I20240506 11:40:42.106539 17991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:33737 every 8 connection(s)
I20240506 11:40:42.107653 17865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:40:42.112670 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 17865
I20240506 11:40:42.112885 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:40:42.115775 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:38885
--builtin_ntp_servers=127.10.45.148:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20240506 11:40:42.119628 17992 heartbeater.cc:347] Connected to a master server at 127.10.45.190:38885
I20240506 11:40:42.119879 17992 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:42.120491 17992 heartbeater.cc:511] Master 127.10.45.190:38885 requested a full tablet report, sending...
I20240506 11:40:42.123232 17806 ts_manager.cc:194] Registered new tserver with Master: e17b56ac16374923a53f885332de43e5 (127.10.45.129:33737)
I20240506 11:40:42.124567 17806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:40497
W20240506 11:40:42.414073 17995 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20240506 11:40:42.414413 17995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:42.414500 17995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:42.414620 17995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:42.421355 17995 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20240506 11:40:42.421439 17995 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20240506 11:40:42.421531 17995 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240506 11:40:42.421620 17995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:42.421795 17995 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20240506 11:40:42.421873 17995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:40:42.429977 17995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:38885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:42.431301 17995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:42.432952 17995 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:42.444662 18001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:42.445503 18003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:42.446357 18000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:42.447279 17995 server_base.cc:1034] running on GCE node
I20240506 11:40:42.447748 17995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:42.448534 17995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:42.449715 17995 hybrid_clock.cc:648] HybridClock initialized: now 1714995642449681 us; error 31 us; skew 500 ppm
I20240506 11:40:42.449970 17995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:42.451995 17995 webserver.cc:460] Webserver started at http://127.10.45.130:37069/ using document root <none> and password file <none>
I20240506 11:40:42.452665 17995 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:42.452769 17995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:42.453042 17995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:42.455513 17995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c3a9e751443445a791bcfaf771bc42b9"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:42.456279 17995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c3a9e751443445a791bcfaf771bc42b9"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:42.460904 17995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240506 11:40:42.463851 18009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:42.465005 17995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240506 11:40:42.465147 17995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c3a9e751443445a791bcfaf771bc42b9"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:42.465315 17995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:42.479494 17995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:42.480104 17995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:42.480348 17995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:42.481150 17995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:42.482493 17995 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:42.482586 17995 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:42.482689 17995 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:42.482765 17995 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:42.512805 17995 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:36871
I20240506 11:40:42.512846 18121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:36871 every 8 connection(s)
I20240506 11:40:42.513924 17995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:40:42.515272 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 17995
I20240506 11:40:42.515435 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:40:42.518353 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:38885
--builtin_ntp_servers=127.10.45.148:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20240506 11:40:42.525617 18122 heartbeater.cc:347] Connected to a master server at 127.10.45.190:38885
I20240506 11:40:42.525902 18122 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:42.526549 18122 heartbeater.cc:511] Master 127.10.45.190:38885 requested a full tablet report, sending...
I20240506 11:40:42.527743 17806 ts_manager.cc:194] Registered new tserver with Master: c3a9e751443445a791bcfaf771bc42b9 (127.10.45.130:36871)
I20240506 11:40:42.528439 17806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:56875
W20240506 11:40:42.810711 18125 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20240506 11:40:42.811002 18125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:42.811061 18125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:42.811127 18125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:42.817538 18125 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20240506 11:40:42.817606 18125 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20240506 11:40:42.817667 18125 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240506 11:40:42.817746 18125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:42.817848 18125 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20240506 11:40:42.817896 18125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:40:42.825788 18125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:38885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:42.826833 18125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:42.828367 18125 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:42.839589 18130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:42.840473 18133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:42.841300 18131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:42.842105 18125 server_base.cc:1034] running on GCE node
I20240506 11:40:42.842631 18125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:42.843465 18125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:42.844645 18125 hybrid_clock.cc:648] HybridClock initialized: now 1714995642844603 us; error 41 us; skew 500 ppm
I20240506 11:40:42.844921 18125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:42.852694 18125 webserver.cc:460] Webserver started at http://127.10.45.131:45463/ using document root <none> and password file <none>
I20240506 11:40:42.853592 18125 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:42.853737 18125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:42.854064 18125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:42.856755 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "842d390c25fb42329cf09feba4f2166f"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:42.857503 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "842d390c25fb42329cf09feba4f2166f"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:42.861755 18125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240506 11:40:42.864768 18139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:42.865962 18125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 11:40:42.866098 18125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "842d390c25fb42329cf09feba4f2166f"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:42.866264 18125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:42.880905 18125 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:42.881552 18125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:42.881835 18125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:42.882694 18125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:42.884004 18125 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:42.884093 18125 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:42.884199 18125 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:42.884274 18125 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:42.913236 18125 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:44391
I20240506 11:40:42.913292 18251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:44391 every 8 connection(s)
I20240506 11:40:42.914377 18125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:40:42.918462 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 18125
I20240506 11:40:42.918607 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:40:42.925537 18252 heartbeater.cc:347] Connected to a master server at 127.10.45.190:38885
I20240506 11:40:42.925791 18252 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:42.926424 18252 heartbeater.cc:511] Master 127.10.45.190:38885 requested a full tablet report, sending...
I20240506 11:40:42.927479 17806 ts_manager.cc:194] Registered new tserver with Master: 842d390c25fb42329cf09feba4f2166f (127.10.45.131:44391)
I20240506 11:40:42.928150 17806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:50553
I20240506 11:40:42.933601 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:40:42.946238 17806 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59280:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240506 11:40:42.985630 18187 tablet_service.cc:1467] Processing CreateTablet for tablet b0da247a05c048548395d903376c4da2 (DEFAULT_TABLE table=TestTable [id=9783bb3cff46465983ecd2d49219f037]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:42.985854 18057 tablet_service.cc:1467] Processing CreateTablet for tablet b0da247a05c048548395d903376c4da2 (DEFAULT_TABLE table=TestTable [id=9783bb3cff46465983ecd2d49219f037]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:42.986192 17927 tablet_service.cc:1467] Processing CreateTablet for tablet b0da247a05c048548395d903376c4da2 (DEFAULT_TABLE table=TestTable [id=9783bb3cff46465983ecd2d49219f037]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:42.986855 18187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0da247a05c048548395d903376c4da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:42.987118 18057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0da247a05c048548395d903376c4da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:42.987324 17927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0da247a05c048548395d903376c4da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:42.995366 18270 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Bootstrap starting.
I20240506 11:40:42.998143 18270 tablet_bootstrap.cc:654] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:42.998677 18271 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: Bootstrap starting.
I20240506 11:40:42.999397 18272 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Bootstrap starting.
I20240506 11:40:42.999397 18270 log.cc:824] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:43.001986 18270 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: No bootstrap required, opened a new log
I20240506 11:40:43.002215 18270 ts_tablet_manager.cc:1367] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240506 11:40:43.002365 18272 tablet_bootstrap.cc:654] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:43.003165 18271 tablet_bootstrap.cc:654] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:43.003438 18272 log.cc:824] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:43.004231 18271 log.cc:824] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:43.005730 18272 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: No bootstrap required, opened a new log
I20240506 11:40:43.006014 18272 ts_tablet_manager.cc:1367] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240506 11:40:43.006572 18271 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: No bootstrap required, opened a new log
I20240506 11:40:43.006796 18271 ts_tablet_manager.cc:1367] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240506 11:40:43.008076 18270 raft_consensus.cc:348] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } }
I20240506 11:40:43.008365 18270 raft_consensus.cc:374] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:43.008463 18270 raft_consensus.cc:729] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842d390c25fb42329cf09feba4f2166f, State: Initialized, Role: FOLLOWER
I20240506 11:40:43.008984 18270 consensus_queue.cc:260] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } }
I20240506 11:40:43.009922 18252 heartbeater.cc:503] Master 127.10.45.190:38885 was elected leader, sending a full tablet report...
I20240506 11:40:43.010309 18270 ts_tablet_manager.cc:1398] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Time spent starting tablet: real 0.008s user 0.004s sys 0.005s
I20240506 11:40:43.013226 18272 raft_consensus.cc:348] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } }
I20240506 11:40:43.013551 18272 raft_consensus.cc:374] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:43.013705 18272 raft_consensus.cc:729] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e17b56ac16374923a53f885332de43e5, State: Initialized, Role: FOLLOWER
I20240506 11:40:43.014266 18272 consensus_queue.cc:260] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } }
I20240506 11:40:43.015219 17992 heartbeater.cc:503] Master 127.10.45.190:38885 was elected leader, sending a full tablet report...
W20240506 11:40:43.015796 18253 tablet.cc:2369] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:43.015709 18271 raft_consensus.cc:348] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } }
I20240506 11:40:43.016070 18271 raft_consensus.cc:374] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:43.016186 18271 raft_consensus.cc:729] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a9e751443445a791bcfaf771bc42b9, State: Initialized, Role: FOLLOWER
I20240506 11:40:43.016726 18271 consensus_queue.cc:260] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } }
I20240506 11:40:43.016912 18272 ts_tablet_manager.cc:1398] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240506 11:40:43.017624 18122 heartbeater.cc:503] Master 127.10.45.190:38885 was elected leader, sending a full tablet report...
I20240506 11:40:43.018113 18271 ts_tablet_manager.cc:1398] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240506 11:40:43.092911 18277 raft_consensus.cc:482] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:40:43.093156 18277 raft_consensus.cc:504] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } }
I20240506 11:40:43.094740 18277 leader_election.cc:288] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3a9e751443445a791bcfaf771bc42b9 (127.10.45.130:36871), 842d390c25fb42329cf09feba4f2166f (127.10.45.131:44391)
I20240506 11:40:43.101089 18207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da247a05c048548395d903376c4da2" candidate_uuid: "e17b56ac16374923a53f885332de43e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842d390c25fb42329cf09feba4f2166f" is_pre_election: true
I20240506 11:40:43.101177 18077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da247a05c048548395d903376c4da2" candidate_uuid: "e17b56ac16374923a53f885332de43e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a9e751443445a791bcfaf771bc42b9" is_pre_election: true
I20240506 11:40:43.101511 18207 raft_consensus.cc:2454] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e17b56ac16374923a53f885332de43e5 in term 0.
I20240506 11:40:43.101572 18077 raft_consensus.cc:2454] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e17b56ac16374923a53f885332de43e5 in term 0.
I20240506 11:40:43.102023 17882 leader_election.cc:302] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 842d390c25fb42329cf09feba4f2166f, e17b56ac16374923a53f885332de43e5; no voters:
I20240506 11:40:43.102380 18277 raft_consensus.cc:2781] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 11:40:43.102519 18277 raft_consensus.cc:482] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:40:43.102629 18277 raft_consensus.cc:3037] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:43.104462 18277 raft_consensus.cc:504] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } }
I20240506 11:40:43.104930 18277 leader_election.cc:288] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [CANDIDATE]: Term 1 election: Requested vote from peers c3a9e751443445a791bcfaf771bc42b9 (127.10.45.130:36871), 842d390c25fb42329cf09feba4f2166f (127.10.45.131:44391)
I20240506 11:40:43.105271 18077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da247a05c048548395d903376c4da2" candidate_uuid: "e17b56ac16374923a53f885332de43e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a9e751443445a791bcfaf771bc42b9"
I20240506 11:40:43.105342 18207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da247a05c048548395d903376c4da2" candidate_uuid: "e17b56ac16374923a53f885332de43e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842d390c25fb42329cf09feba4f2166f"
I20240506 11:40:43.105424 18077 raft_consensus.cc:3037] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:43.105520 18207 raft_consensus.cc:3037] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:43.107280 18077 raft_consensus.cc:2454] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e17b56ac16374923a53f885332de43e5 in term 1.
I20240506 11:40:43.107602 17880 leader_election.cc:302] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3a9e751443445a791bcfaf771bc42b9, e17b56ac16374923a53f885332de43e5; no voters:
I20240506 11:40:43.107867 18277 raft_consensus.cc:2781] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:43.108124 18207 raft_consensus.cc:2454] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e17b56ac16374923a53f885332de43e5 in term 1.
I20240506 11:40:43.108350 18277 raft_consensus.cc:686] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 LEADER]: Becoming Leader. State: Replica: e17b56ac16374923a53f885332de43e5, State: Running, Role: LEADER
I20240506 11:40:43.108872 18277 consensus_queue.cc:237] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } }
W20240506 11:40:43.110805 17993 tablet.cc:2369] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:43.113220 17805 catalog_manager.cc:5484] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e17b56ac16374923a53f885332de43e5 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "e17b56ac16374923a53f885332de43e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } } }
W20240506 11:40:43.116508 18123 tablet.cc:2369] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:43.154700 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:40:43.156679 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e17b56ac16374923a53f885332de43e5 to finish bootstrapping
I20240506 11:40:43.164062 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c3a9e751443445a791bcfaf771bc42b9 to finish bootstrapping
I20240506 11:40:43.170048 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 842d390c25fb42329cf09feba4f2166f to finish bootstrapping
I20240506 11:40:43.192318 10422 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20240506 11:40:43.295661 10422 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20240506 11:40:43.498587 10422 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20240506 11:40:43.581475 18277 consensus_queue.cc:1035] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:43.588184 18277 consensus_queue.cc:1035] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:43.816541 10422 test_util.cc:268] Using random seed: -1977345492
I20240506 11:40:43.828944 10422 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
I20240506 11:40:43.830250 10422 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
I20240506 11:40:43.914489 17993 maintenance_manager.cc:419] P e17b56ac16374923a53f885332de43e5: Scheduling FlushMRSOp(b0da247a05c048548395d903376c4da2): 1840872 bytes log retention, and flush 2998537 bytes memory
I20240506 11:40:43.920238 18123 maintenance_manager.cc:419] P c3a9e751443445a791bcfaf771bc42b9: Scheduling FlushMRSOp(b0da247a05c048548395d903376c4da2): 1577888 bytes log retention, and flush 2998537 bytes memory
I20240506 11:40:44.053579 10422 raft_consensus-itest-base.cc:274] Waiting for log GC on e17b56ac16374923a53f885332de43e5
I20240506 11:40:44.077339 18014 maintenance_manager.cc:643] P c3a9e751443445a791bcfaf771bc42b9: FlushMRSOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.156s user 0.061s sys 0.015s Metrics: {"bytes_written":135869,"cfile_init":1,"compiler_manager_pool.queue_time_us":449,"dirs.queue_time_us":4260,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1109,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1060,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":17,"thread_start_us":8856,"threads_started":2}
I20240506 11:40:44.077966 17884 maintenance_manager.cc:643] P e17b56ac16374923a53f885332de43e5: FlushMRSOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.162s user 0.064s sys 0.023s Metrics: {"bytes_written":135803,"cfile_init":1,"compiler_manager_pool.queue_time_us":473,"dirs.queue_time_us":20658,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":837,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1019,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":16,"thread_start_us":20880,"threads_started":2}
I20240506 11:40:44.078938 18123 maintenance_manager.cc:419] P c3a9e751443445a791bcfaf771bc42b9: Scheduling LogGCOp(b0da247a05c048548395d903376c4da2): free 2235424 bytes of WAL
I20240506 11:40:44.079398 18014 log_reader.cc:386] T b0da247a05c048548395d903376c4da2: removed 3 log segments from log reader
I20240506 11:40:44.079486 17993 maintenance_manager.cc:419] P e17b56ac16374923a53f885332de43e5: Scheduling LogGCOp(b0da247a05c048548395d903376c4da2): free 1840872 bytes of WAL
I20240506 11:40:44.079572 18014 log.cc:1076] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: Deleting log segment in path: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/wals/b0da247a05c048548395d903376c4da2/wal-000000001 (ops 1-6)
I20240506 11:40:44.079747 18014 log.cc:1076] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: Deleting log segment in path: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/wals/b0da247a05c048548395d903376c4da2/wal-000000002 (ops 7-13)
I20240506 11:40:44.079866 17884 log_reader.cc:386] T b0da247a05c048548395d903376c4da2: removed 2 log segments from log reader
I20240506 11:40:44.079888 18014 log.cc:1076] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: Deleting log segment in path: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/wals/b0da247a05c048548395d903376c4da2/wal-000000003 (ops 14-18)
I20240506 11:40:44.080019 17884 log.cc:1076] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Deleting log segment in path: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/b0da247a05c048548395d903376c4da2/wal-000000001 (ops 1-8)
I20240506 11:40:44.080155 17884 log.cc:1076] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Deleting log segment in path: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/b0da247a05c048548395d903376c4da2/wal-000000002 (ops 9-15)
I20240506 11:40:44.080991 18014 maintenance_manager.cc:643] P c3a9e751443445a791bcfaf771bc42b9: LogGCOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {}
I20240506 11:40:44.081027 17884 maintenance_manager.cc:643] P e17b56ac16374923a53f885332de43e5: LogGCOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240506 11:40:44.081435 18123 maintenance_manager.cc:419] P c3a9e751443445a791bcfaf771bc42b9: Scheduling FlushMRSOp(b0da247a05c048548395d903376c4da2): 2103901 bytes log retention, and flush 4038921 bytes memory
I20240506 11:40:44.081446 17993 maintenance_manager.cc:419] P e17b56ac16374923a53f885332de43e5: Scheduling FlushMRSOp(b0da247a05c048548395d903376c4da2): 2761368 bytes log retention, and flush 4038921 bytes memory
I20240506 11:40:44.084815 10422 raft_consensus-itest-base.cc:279] Log GC complete on e17b56ac16374923a53f885332de43e5
I20240506 11:40:44.140987 18014 maintenance_manager.cc:643] P c3a9e751443445a791bcfaf771bc42b9: FlushMRSOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.059s user 0.051s sys 0.004s Metrics: {"bytes_written":135869,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1568,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":878,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":20}
I20240506 11:40:44.141808 17884 maintenance_manager.cc:643] P e17b56ac16374923a53f885332de43e5: FlushMRSOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.060s user 0.055s sys 0.001s Metrics: {"bytes_written":135869,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1055,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":828,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":21}
I20240506 11:40:44.142393 18123 maintenance_manager.cc:419] P c3a9e751443445a791bcfaf771bc42b9: Scheduling LogGCOp(b0da247a05c048548395d903376c4da2): free 788953 bytes of WAL
I20240506 11:40:44.142768 18014 log_reader.cc:386] T b0da247a05c048548395d903376c4da2: removed 1 log segments from log reader
I20240506 11:40:44.142903 18014 log.cc:1076] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9: Deleting log segment in path: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/wals/b0da247a05c048548395d903376c4da2/wal-000000004 (ops 19-24)
I20240506 11:40:44.143016 17993 maintenance_manager.cc:419] P e17b56ac16374923a53f885332de43e5: Scheduling LogGCOp(b0da247a05c048548395d903376c4da2): free 920494 bytes of WAL
I20240506 11:40:44.143296 17884 log_reader.cc:386] T b0da247a05c048548395d903376c4da2: removed 1 log segments from log reader
I20240506 11:40:44.143409 17884 log.cc:1076] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Deleting log segment in path: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/b0da247a05c048548395d903376c4da2/wal-000000003 (ops 16-22)
I20240506 11:40:44.143419 18014 maintenance_manager.cc:643] P c3a9e751443445a791bcfaf771bc42b9: LogGCOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {}
I20240506 11:40:44.143795 18123 maintenance_manager.cc:419] P c3a9e751443445a791bcfaf771bc42b9: Scheduling UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2): 685 bytes on disk
I20240506 11:40:44.143846 17884 maintenance_manager.cc:643] P e17b56ac16374923a53f885332de43e5: LogGCOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240506 11:40:44.144145 17993 maintenance_manager.cc:419] P e17b56ac16374923a53f885332de43e5: Scheduling UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2): 685 bytes on disk
I20240506 11:40:44.144918 18014 maintenance_manager.cc:643] P c3a9e751443445a791bcfaf771bc42b9: UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":156,"lbm_reads_lt_1ms":8}
I20240506 11:40:44.145013 17884 maintenance_manager.cc:643] P e17b56ac16374923a53f885332de43e5: UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":149,"lbm_reads_lt_1ms":8}
I20240506 11:40:44.145638 18123 maintenance_manager.cc:419] P c3a9e751443445a791bcfaf771bc42b9: Scheduling CompactRowSetsOp(b0da247a05c048548395d903376c4da2): perf score=0.993908
I20240506 11:40:44.145660 17993 maintenance_manager.cc:419] P e17b56ac16374923a53f885332de43e5: Scheduling CompactRowSetsOp(b0da247a05c048548395d903376c4da2): perf score=0.993913
I20240506 11:40:44.259419 18014 maintenance_manager.cc:643] P c3a9e751443445a791bcfaf771bc42b9: CompactRowSetsOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.114s user 0.098s sys 0.013s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264438,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1962,"dirs.run_cpu_time_us":905,"dirs.run_wall_time_us":5332,"drs_written":1,"lbm_read_time_us":1108,"lbm_reads_lt_1ms":46,"lbm_write_time_us":945,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":224,"rows_written":37,"thread_start_us":8875,"threads_started":2}
I20240506 11:40:44.260427 18123 maintenance_manager.cc:419] P c3a9e751443445a791bcfaf771bc42b9: Scheduling UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2): 550 bytes on disk
I20240506 11:40:44.261114 18014 maintenance_manager.cc:643] P c3a9e751443445a791bcfaf771bc42b9: UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240506 11:40:44.261997 17884 maintenance_manager.cc:643] P e17b56ac16374923a53f885332de43e5: CompactRowSetsOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.116s user 0.092s sys 0.019s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264310,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":2387,"dirs.run_cpu_time_us":890,"dirs.run_wall_time_us":4733,"drs_written":1,"lbm_read_time_us":1097,"lbm_reads_lt_1ms":46,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"mutex_wait_us":115,"num_input_rowsets":2,"peak_mem_usage":208,"rows_written":37,"thread_start_us":10685,"threads_started":2}
I20240506 11:40:44.262794 17993 maintenance_manager.cc:419] P e17b56ac16374923a53f885332de43e5: Scheduling UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2): 550 bytes on disk
I20240506 11:40:44.263403 17884 maintenance_manager.cc:643] P e17b56ac16374923a53f885332de43e5: UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240506 11:40:46.086295 10422 raft_consensus-itest-base.cc:295] Servers converged with original term 1
I20240506 11:40:46.086433 10422 raft_consensus-itest-base.cc:300] Resuming 842d390c25fb42329cf09feba4f2166f
I20240506 11:40:46.087424 18289 raft_consensus.cc:482] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e17b56ac16374923a53f885332de43e5)
I20240506 11:40:46.087641 18289 raft_consensus.cc:504] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } }
I20240506 11:40:46.089406 18289 leader_election.cc:288] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3a9e751443445a791bcfaf771bc42b9 (127.10.45.130:36871), e17b56ac16374923a53f885332de43e5 (127.10.45.129:33737)
I20240506 11:40:46.101717 18324 consensus_queue.cc:740] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [LEADER]: The logs necessary to catch up peer 842d390c25fb42329cf09feba4f2166f have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 3..32: Segment 1 which contained index 3 has been GCed)
I20240506 11:40:46.107715 17947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da247a05c048548395d903376c4da2" candidate_uuid: "842d390c25fb42329cf09feba4f2166f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e17b56ac16374923a53f885332de43e5" is_pre_election: true
I20240506 11:40:46.108058 18077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da247a05c048548395d903376c4da2" candidate_uuid: "842d390c25fb42329cf09feba4f2166f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c3a9e751443445a791bcfaf771bc42b9" is_pre_election: true
I20240506 11:40:46.108733 18140 leader_election.cc:302] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 842d390c25fb42329cf09feba4f2166f; no voters: c3a9e751443445a791bcfaf771bc42b9, e17b56ac16374923a53f885332de43e5
I20240506 11:40:46.109100 18289 raft_consensus.cc:2726] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 11:40:46.109930 18331 raft_consensus.cc:998] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Attempting to remove follower 842d390c25fb42329cf09feba4f2166f from the Raft config. Reason: The logs necessary to catch up peer 842d390c25fb42329cf09feba4f2166f have been garbage collected. The replica will never be able to catch up
I20240506 11:40:46.111182 18331 consensus_queue.cc:237] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } }
I20240506 11:40:46.113129 18077 raft_consensus.cc:1261] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [term 1 FOLLOWER]: Refusing update from remote peer e17b56ac16374923a53f885332de43e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240506 11:40:46.113660 18324 consensus_queue.cc:1035] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20240506 11:40:46.116832 18331 raft_consensus.cc:2932] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 842d390c25fb42329cf09feba4f2166f (127.10.45.131) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } }
I20240506 11:40:46.117399 18077 raft_consensus.cc:2932] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 842d390c25fb42329cf09feba4f2166f (127.10.45.131) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } }
I20240506 11:40:46.121284 17804 catalog_manager.cc:5484] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 reported cstate change: config changed from index -1 to 39, VOTER 842d390c25fb42329cf09feba4f2166f (127.10.45.131) evicted. New cstate: current_term: 1 leader_uuid: "e17b56ac16374923a53f885332de43e5" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } }
I20240506 11:40:46.125734 18187 tablet_service.cc:1514] Processing DeleteTablet for tablet b0da247a05c048548395d903376c4da2 with delete_type TABLET_DATA_TOMBSTONED (TS 842d390c25fb42329cf09feba4f2166f not found in new config with opid_index 39) from {username='slave'} at 127.0.0.1:35230
I20240506 11:40:46.126508 18339 tablet_replica.cc:330] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: stopping tablet replica
I20240506 11:40:46.126859 18339 raft_consensus.cc:2229] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240506 11:40:46.127136 18339 pending_rounds.cc:62] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Trying to abort 1 pending ops.
I20240506 11:40:46.127250 18339 pending_rounds.cc:68] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7024622157202063360 op_type: WRITE_OP write_request { tablet_id: "b0da247a05c048548395d903376c4da2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\024\237P \371\350y\276\000\000\000\000\000\000\000\000\000\000\002\000\000\000\000\000" indirect_data: "<redacted>""000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
I20240506 11:40:46.129462 18339 raft_consensus.cc:2258] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:40:46.130724 18339 ts_tablet_manager.cc:1875] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:40:46.134675 18339 ts_tablet_manager.cc:1888] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240506 11:40:46.134802 18339 log.cc:1196] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/wals/b0da247a05c048548395d903376c4da2
I20240506 11:40:46.135617 17792 catalog_manager.cc:4830] TS 842d390c25fb42329cf09feba4f2166f (127.10.45.131:44391): tablet b0da247a05c048548395d903376c4da2 (table TestTable [id=9783bb3cff46465983ecd2d49219f037]) successfully deleted
I20240506 11:40:46.140738 18247 generic_service.cc:175] {username='slave'} at 127.0.0.1:35238 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0'
I20240506 11:40:46.142370 17947 consensus_queue.cc:237] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } }
I20240506 11:40:46.144160 18077 raft_consensus.cc:1261] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [term 1 FOLLOWER]: Refusing update from remote peer e17b56ac16374923a53f885332de43e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
W20240506 11:40:46.144562 17882 consensus_peers.cc:485] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 -> Peer 842d390c25fb42329cf09feba4f2166f (127.10.45.131:44391): Couldn't send request to peer 842d390c25fb42329cf09feba4f2166f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240506 11:40:46.144685 18331 consensus_queue.cc:1035] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20240506 11:40:46.146526 18324 raft_consensus.cc:2932] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index 39 to 40, VOTER 842d390c25fb42329cf09feba4f2166f (127.10.45.131) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } } }
I20240506 11:40:46.147058 18077 raft_consensus.cc:2932] T b0da247a05c048548395d903376c4da2 P c3a9e751443445a791bcfaf771bc42b9 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index 39 to 40, VOTER 842d390c25fb42329cf09feba4f2166f (127.10.45.131) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } } }
I20240506 11:40:46.150326 17804 catalog_manager.cc:5484] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 reported cstate change: config changed from index 39 to 40, VOTER 842d390c25fb42329cf09feba4f2166f (127.10.45.131) added. New cstate: current_term: 1 leader_uuid: "e17b56ac16374923a53f885332de43e5" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } } }
I20240506 11:40:46.717029 18341 ts_tablet_manager.cc:902] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Initiating tablet copy from peer e17b56ac16374923a53f885332de43e5 (127.10.45.129:33737)
I20240506 11:40:46.717804 18341 tablet_copy_client.cc:316] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: tablet copy: Beginning tablet copy session from remote peer at address 127.10.45.129:33737
I20240506 11:40:46.718696 17967 tablet_copy_service.cc:140] P e17b56ac16374923a53f885332de43e5: Received BeginTabletCopySession request for tablet b0da247a05c048548395d903376c4da2 from peer 842d390c25fb42329cf09feba4f2166f ({username='slave'} at 127.10.45.131:47241)
I20240506 11:40:46.718888 17967 tablet_copy_service.cc:161] P e17b56ac16374923a53f885332de43e5: Beginning new tablet copy session on tablet b0da247a05c048548395d903376c4da2 from peer 842d390c25fb42329cf09feba4f2166f at {username='slave'} at 127.10.45.131:47241: session id = 842d390c25fb42329cf09feba4f2166f-b0da247a05c048548395d903376c4da2
I20240506 11:40:46.721652 17967 tablet_copy_source_session.cc:205] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Tablet Copy: opened 5 blocks and 3 log segments
I20240506 11:40:46.723639 18341 ts_tablet_manager.cc:1875] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240506 11:40:46.727996 18341 ts_tablet_manager.cc:1888] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20240506 11:40:46.728225 18341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0da247a05c048548395d903376c4da2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240506 11:40:46.730556 18341 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240506 11:40:46.730654 18341 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20240506 11:40:46.730712 18341 fault_injection.cc:59] injecting fault for kudu (pid 18125): FLAGS_tablet_copy_fault_crash_on_fetch_all (process will exit)
I20240506 11:40:46.811667 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 17865
I20240506 11:40:46.812868 17987 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:46.922740 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 17865
I20240506 11:40:46.946416 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 17995
I20240506 11:40:46.947424 18117 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:47.056090 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 17995
I20240506 11:40:47.076537 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 17775
I20240506 11:40:47.077546 17836 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:47.169684 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 17775
I20240506 11:40:47.191982 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:38885
--webserver_interface=127.10.45.190
--webserver_port=34349
--builtin_ntp_servers=127.10.45.148:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:38885
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20240506 11:40:47.499161 18346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:47.499452 18346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:47.499579 18346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:47.506183 18346 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240506 11:40:47.506327 18346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:40:47.506398 18346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:47.506482 18346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:40:47.506547 18346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:40:47.514815 18346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:38885
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:38885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=34349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:47.515930 18346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:47.517437 18346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:47.527626 18351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:47.528491 18352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:47.529155 18354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:47.529914 18346 server_base.cc:1034] running on GCE node
I20240506 11:40:47.530503 18346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:47.531740 18346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:47.532924 18346 hybrid_clock.cc:648] HybridClock initialized: now 1714995647532887 us; error 30 us; skew 500 ppm
I20240506 11:40:47.533215 18346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:47.535151 18346 webserver.cc:460] Webserver started at http://127.10.45.190:34349/ using document root <none> and password file <none>
I20240506 11:40:47.535789 18346 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:47.535915 18346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:47.540460 18346 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240506 11:40:47.543148 18360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:47.544283 18346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240506 11:40:47.544428 18346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1593b0c3f63d4da38dbb73dc02c60e24"
format_stamp: "Formatted at 2024-05-06 11:40:41 on dist-test-slave-7l0g"
I20240506 11:40:47.545118 18346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:47.583314 18346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:47.584148 18346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:47.584434 18346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:47.601181 18346 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:38885
I20240506 11:40:47.601214 18411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:38885 every 8 connection(s)
I20240506 11:40:47.602506 18346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:40:47.608178 18412 sys_catalog.cc:263] Verifying existing consensus state
I20240506 11:40:47.610836 18412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24: Bootstrap starting.
I20240506 11:40:47.612234 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 18346
I20240506 11:40:47.613353 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:33737
--local_ip_for_outbound_sockets=127.10.45.129
--tserver_master_addrs=127.10.45.190:38885
--webserver_port=37269
--webserver_interface=127.10.45.129
--builtin_ntp_servers=127.10.45.148:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20240506 11:40:47.628465 18412 log.cc:824] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:47.639173 18412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240506 11:40:47.639596 18412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24: Bootstrap complete.
I20240506 11:40:47.646551 18412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } }
I20240506 11:40:47.647218 18412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1593b0c3f63d4da38dbb73dc02c60e24, State: Initialized, Role: FOLLOWER
I20240506 11:40:47.647760 18412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } }
I20240506 11:40:47.647941 18412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:40:47.648051 18412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:40:47.648193 18412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:47.651437 18412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } }
I20240506 11:40:47.651845 18412 leader_election.cc:302] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1593b0c3f63d4da38dbb73dc02c60e24; no voters:
I20240506 11:40:47.652441 18412 leader_election.cc:288] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240506 11:40:47.652552 18416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 11:40:47.654664 18412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:40:47.659446 18416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [term 2 LEADER]: Becoming Leader. State: Replica: 1593b0c3f63d4da38dbb73dc02c60e24, State: Running, Role: LEADER
I20240506 11:40:47.660285 18416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } }
I20240506 11:40:47.663421 18417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } } }
I20240506 11:40:47.663786 18417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:47.664294 18418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1593b0c3f63d4da38dbb73dc02c60e24. Latest consensus state: current_term: 2 leader_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1593b0c3f63d4da38dbb73dc02c60e24" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 38885 } } }
I20240506 11:40:47.664531 18418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:47.666724 18430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:40:47.672320 18430 catalog_manager.cc:669] Loaded metadata for table TestTable [id=9783bb3cff46465983ecd2d49219f037]
I20240506 11:40:47.674610 18430 tablet_loader.cc:96] loaded metadata for tablet b0da247a05c048548395d903376c4da2 (table TestTable [id=9783bb3cff46465983ecd2d49219f037])
I20240506 11:40:47.674997 18430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:40:47.676342 18430 catalog_manager.cc:1259] Loaded cluster ID: cf2768c9620346fabd342c9bf862f297
I20240506 11:40:47.676436 18430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:40:47.680270 18430 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:40:47.681654 18430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1593b0c3f63d4da38dbb73dc02c60e24: Loaded TSK: 0
I20240506 11:40:47.683022 18430 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240506 11:40:47.924080 18414 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20240506 11:40:47.924436 18414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:47.924530 18414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:47.924643 18414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:47.931901 18414 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20240506 11:40:47.931989 18414 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20240506 11:40:47.932106 18414 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240506 11:40:47.932189 18414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:47.932353 18414 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20240506 11:40:47.932428 18414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:40:47.941182 18414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:33737
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=37269
--tserver_master_addrs=127.10.45.190:38885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:47.942458 18414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:47.944170 18414 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240506 11:40:47.958352 18414 server_base.cc:1034] running on GCE node
W20240506 11:40:47.959141 18439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:47.959854 18442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:47.959894 18440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:47.960763 18414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:47.961870 18414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:47.963070 18414 hybrid_clock.cc:648] HybridClock initialized: now 1714995647962983 us; error 47 us; skew 500 ppm
I20240506 11:40:47.963543 18414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:47.971477 18414 webserver.cc:460] Webserver started at http://127.10.45.129:37269/ using document root <none> and password file <none>
I20240506 11:40:47.972311 18414 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:47.972461 18414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:47.979116 18414 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.004s
I20240506 11:40:47.994256 18450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:47.995466 18414 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.000s
I20240506 11:40:47.995630 18414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e17b56ac16374923a53f885332de43e5"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:47.996392 18414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 5
Total live bytes: 136551
Total live bytes (after alignment): 155648
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:48.022899 18414 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:48.023694 18414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:48.023947 18414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:48.024780 18414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:48.026546 18457 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240506 11:40:48.031911 18414 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 11:40:48.031989 18414 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240506 11:40:48.032109 18414 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 11:40:48.034548 18414 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 11:40:48.034636 18414 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240506 11:40:48.034961 18457 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Bootstrap starting.
I20240506 11:40:48.054244 18457 log.cc:824] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:48.070479 18457 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=4 ignored=4} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=1. Pending: 3 replicates
I20240506 11:40:48.072124 18414 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:33737
I20240506 11:40:48.072234 18564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:33737 every 8 connection(s)
I20240506 11:40:48.073462 18414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:40:48.076814 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 18414
I20240506 11:40:48.077818 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:44391
--local_ip_for_outbound_sockets=127.10.45.131
--tserver_master_addrs=127.10.45.190:38885
--webserver_port=45463
--webserver_interface=127.10.45.131
--builtin_ntp_servers=127.10.45.148:40329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20240506 11:40:48.080024 18457 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=12 ignored=12} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=1. Pending: 2 replicates
I20240506 11:40:48.085619 18457 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Bootstrap replayed 3/3 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=16} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=1. Pending: 0 replicates
I20240506 11:40:48.086365 18457 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Bootstrap complete.
I20240506 11:40:48.091168 18457 ts_tablet_manager.cc:1367] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Time spent bootstrapping tablet: real 0.057s user 0.031s sys 0.019s
I20240506 11:40:48.098471 18457 raft_consensus.cc:348] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } }
I20240506 11:40:48.099812 18457 raft_consensus.cc:729] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e17b56ac16374923a53f885332de43e5, State: Initialized, Role: FOLLOWER
I20240506 11:40:48.100611 18457 consensus_queue.cc:260] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } }
I20240506 11:40:48.101380 18565 heartbeater.cc:347] Connected to a master server at 127.10.45.190:38885
I20240506 11:40:48.101655 18565 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:48.103765 18565 heartbeater.cc:511] Master 127.10.45.190:38885 requested a full tablet report, sending...
I20240506 11:40:48.107605 18377 ts_manager.cc:194] Registered new tserver with Master: e17b56ac16374923a53f885332de43e5 (127.10.45.129:33737)
I20240506 11:40:48.110816 18457 ts_tablet_manager.cc:1398] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Time spent starting tablet: real 0.019s user 0.011s sys 0.005s
I20240506 11:40:48.112041 18377 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:42291
I20240506 11:40:48.115463 18565 heartbeater.cc:503] Master 127.10.45.190:38885 was elected leader, sending a full tablet report...
I20240506 11:40:48.175865 18566 maintenance_manager.cc:419] P e17b56ac16374923a53f885332de43e5: Scheduling UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2): 550 bytes on disk
I20240506 11:40:48.177206 18455 maintenance_manager.cc:643] P e17b56ac16374923a53f885332de43e5: UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240506 11:40:48.392556 18569 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20240506 11:40:48.392866 18569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:48.392923 18569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:48.392988 18569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:48.399426 18569 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20240506 11:40:48.399489 18569 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20240506 11:40:48.399542 18569 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240506 11:40:48.399590 18569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:48.399694 18569 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20240506 11:40:48.399741 18569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:40:48.407464 18569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:40329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:44391
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=45463
--tserver_master_addrs=127.10.45.190:38885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:48.408502 18569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:48.409981 18569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:48.421288 18578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:48.422202 18575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:48.423084 18576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:48.423935 18569 server_base.cc:1034] running on GCE node
I20240506 11:40:48.424481 18569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:48.425340 18569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:48.426537 18569 hybrid_clock.cc:648] HybridClock initialized: now 1714995648426489 us; error 44 us; skew 500 ppm
I20240506 11:40:48.426817 18569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:48.428921 18569 webserver.cc:460] Webserver started at http://127.10.45.131:45463/ using document root <none> and password file <none>
I20240506 11:40:48.429618 18569 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:48.429736 18569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:48.434593 18569 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240506 11:40:48.437106 18584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:48.438241 18569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240506 11:40:48.438393 18569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "842d390c25fb42329cf09feba4f2166f"
format_stamp: "Formatted at 2024-05-06 11:40:42 on dist-test-slave-7l0g"
I20240506 11:40:48.439183 18569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:48.453771 18569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:48.454420 18569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:48.454658 18569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:48.455536 18569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:48.457188 18591 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
W20240506 11:40:48.461493 18591 ts_tablet_manager.cc:1822] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20240506 11:40:48.463999 18591 ts_tablet_manager.cc:1875] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:40:48.469882 18591 ts_tablet_manager.cc:1888] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240506 11:40:48.470814 18569 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20240506 11:40:48.470906 18569 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20240506 11:40:48.471060 18569 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:48.471117 18569 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:48.500558 18569 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:44391
I20240506 11:40:48.500610 18697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:44391 every 8 connection(s)
I20240506 11:40:48.501883 18569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:40:48.507450 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 18569
I20240506 11:40:48.507722 10422 test_util.cc:268] Using random seed: -1972654276
I20240506 11:40:48.514660 18698 heartbeater.cc:347] Connected to a master server at 127.10.45.190:38885
I20240506 11:40:48.514912 18698 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:48.515648 18698 heartbeater.cc:511] Master 127.10.45.190:38885 requested a full tablet report, sending...
I20240506 11:40:48.517251 18376 ts_manager.cc:194] Registered new tserver with Master: 842d390c25fb42329cf09feba4f2166f (127.10.45.131:44391)
I20240506 11:40:48.518383 18376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:46627
I20240506 11:40:48.523382 10422 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20240506 11:40:48.552505 18479 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.555548 18479 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.556313 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.556908 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.557976 18701 meta_cache.cc:302] tablet b0da247a05c048548395d903376c4da2: replica c3a9e751443445a791bcfaf771bc42b9 (127.10.45.130:36871) has failed: Network error: Client connection negotiation failed: client connection to 127.10.45.130:36871: connect: Connection refused (error 111)
W20240506 11:40:48.574163 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.582401 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.588784 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.594839 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.608095 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.608740 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.618145 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.626284 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.643229 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.653928 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.660934 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.674111 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.689594 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.697566 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.716118 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.735744 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.762403 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.762632 18479 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.783020 18479 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.783351 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.803869 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.825385 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.849054 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.875005 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.903461 18478 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.903625 18479 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.903968 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.957173 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.964069 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:48.994338 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.020680 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.051924 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.083961 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.113461 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.143147 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.156548 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.190454 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.229388 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.266073 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.306880 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.345104 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
W20240506 11:40:49.374323 18477 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39326: Illegal state: replica e17b56ac16374923a53f885332de43e5 is not leader of this config: current role FOLLOWER
I20240506 11:40:49.386598 18714 raft_consensus.cc:482] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:40:49.386845 18714 raft_consensus.cc:504] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } }
I20240506 11:40:49.388384 18714 leader_election.cc:288] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3a9e751443445a791bcfaf771bc42b9 (127.10.45.130:36871), 842d390c25fb42329cf09feba4f2166f (127.10.45.131:44391)
W20240506 11:40:49.390022 18451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.45.130:36871: connect: Connection refused (error 111)
W20240506 11:40:49.391490 18451 leader_election.cc:334] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c3a9e751443445a791bcfaf771bc42b9 (127.10.45.130:36871): Network error: Client connection negotiation failed: client connection to 127.10.45.130:36871: connect: Connection refused (error 111)
I20240506 11:40:49.395051 18653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da247a05c048548395d903376c4da2" candidate_uuid: "e17b56ac16374923a53f885332de43e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "842d390c25fb42329cf09feba4f2166f" is_pre_election: true
I20240506 11:40:49.397102 18653 raft_consensus.cc:1789] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.2
I20240506 11:40:49.397768 18653 raft_consensus.cc:2454] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e17b56ac16374923a53f885332de43e5 in term 1.
I20240506 11:40:49.398226 18453 leader_election.cc:302] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 842d390c25fb42329cf09feba4f2166f, e17b56ac16374923a53f885332de43e5; no voters: c3a9e751443445a791bcfaf771bc42b9
I20240506 11:40:49.398558 18714 raft_consensus.cc:2781] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240506 11:40:49.398665 18714 raft_consensus.cc:482] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:40:49.398772 18714 raft_consensus.cc:3037] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:40:49.401897 18714 raft_consensus.cc:504] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } }
I20240506 11:40:49.402470 18714 leader_election.cc:288] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [CANDIDATE]: Term 2 election: Requested vote from peers c3a9e751443445a791bcfaf771bc42b9 (127.10.45.130:36871), 842d390c25fb42329cf09feba4f2166f (127.10.45.131:44391)
I20240506 11:40:49.402905 18653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0da247a05c048548395d903376c4da2" candidate_uuid: "e17b56ac16374923a53f885332de43e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "842d390c25fb42329cf09feba4f2166f"
I20240506 11:40:49.403118 18653 raft_consensus.cc:1789] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.2
I20240506 11:40:49.403220 18653 raft_consensus.cc:3037] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 1 FOLLOWER]: Advancing to term 2
W20240506 11:40:49.403638 18451 leader_election.cc:334] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c3a9e751443445a791bcfaf771bc42b9 (127.10.45.130:36871): Network error: Client connection negotiation failed: client connection to 127.10.45.130:36871: connect: Connection refused (error 111)
I20240506 11:40:49.405128 18653 raft_consensus.cc:2454] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e17b56ac16374923a53f885332de43e5 in term 2.
I20240506 11:40:49.405463 18453 leader_election.cc:302] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 842d390c25fb42329cf09feba4f2166f, e17b56ac16374923a53f885332de43e5; no voters: c3a9e751443445a791bcfaf771bc42b9
I20240506 11:40:49.405702 18714 raft_consensus.cc:2781] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 2 FOLLOWER]: Leader election won for term 2
I20240506 11:40:49.406026 18714 raft_consensus.cc:686] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [term 2 LEADER]: Becoming Leader. State: Replica: e17b56ac16374923a53f885332de43e5, State: Running, Role: LEADER
I20240506 11:40:49.406551 18714 consensus_queue.cc:237] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } }
I20240506 11:40:49.410311 18377 catalog_manager.cc:5484] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "e17b56ac16374923a53f885332de43e5" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } } }
W20240506 11:40:49.418936 18453 consensus_peers.cc:485] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 -> Peer 842d390c25fb42329cf09feba4f2166f (127.10.45.131:44391): Couldn't send request to peer 842d390c25fb42329cf09feba4f2166f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240506 11:40:49.419272 18451 consensus_peers.cc:485] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5 -> Peer c3a9e751443445a791bcfaf771bc42b9 (127.10.45.130:36871): Couldn't send request to peer c3a9e751443445a791bcfaf771bc42b9. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.130:36871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 11:40:49.520727 18698 heartbeater.cc:503] Master 127.10.45.190:38885 was elected leader, sending a full tablet report...
I20240506 11:40:49.900602 18726 tablet_replica.cc:330] stopping tablet replica
I20240506 11:40:49.900882 18726 raft_consensus.cc:2229] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:40:49.901043 18726 raft_consensus.cc:2258] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:40:49.901320 18726 ts_tablet_manager.cc:902] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Initiating tablet copy from peer e17b56ac16374923a53f885332de43e5 (127.10.45.129:33737)
I20240506 11:40:49.902122 18726 tablet_copy_client.cc:316] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: tablet copy: Beginning tablet copy session from remote peer at address 127.10.45.129:33737
I20240506 11:40:49.908511 18540 tablet_copy_service.cc:140] P e17b56ac16374923a53f885332de43e5: Received BeginTabletCopySession request for tablet b0da247a05c048548395d903376c4da2 from peer 842d390c25fb42329cf09feba4f2166f ({username='slave'} at 127.10.45.131:50111)
I20240506 11:40:49.908710 18540 tablet_copy_service.cc:161] P e17b56ac16374923a53f885332de43e5: Beginning new tablet copy session on tablet b0da247a05c048548395d903376c4da2 from peer 842d390c25fb42329cf09feba4f2166f at {username='slave'} at 127.10.45.131:50111: session id = 842d390c25fb42329cf09feba4f2166f-b0da247a05c048548395d903376c4da2
I20240506 11:40:49.911463 18540 tablet_copy_source_session.cc:205] T b0da247a05c048548395d903376c4da2 P e17b56ac16374923a53f885332de43e5: Tablet Copy: opened 5 blocks and 3 log segments
I20240506 11:40:49.913736 18726 ts_tablet_manager.cc:1875] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240506 11:40:49.917691 18726 ts_tablet_manager.cc:1888] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20240506 11:40:49.918099 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0da247a05c048548395d903376c4da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:49.920648 18726 tablet_copy_client.cc:791] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: tablet copy: Starting download of 5 data blocks...
I20240506 11:40:49.929327 18726 tablet_copy_client.cc:655] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: tablet copy: Starting download of 3 WAL segments...
I20240506 11:40:49.938500 18726 tablet_copy_client.cc:531] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240506 11:40:49.941475 18726 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Bootstrap starting.
I20240506 11:40:49.952000 18726 log.cc:824] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:49.962416 18726 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=4 ignored=4} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240506 11:40:49.969271 18726 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=12 ignored=12} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20240506 11:40:49.973654 18726 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Bootstrap replayed 3/3 log segments. Stats: ops{read=23 overwritten=0 applied=18 ignored=16} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20240506 11:40:49.974251 18726 tablet_bootstrap.cc:492] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Bootstrap complete.
I20240506 11:40:49.975034 18726 ts_tablet_manager.cc:1367] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.020s
I20240506 11:40:49.977558 18726 raft_consensus.cc:348] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 2 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } }
I20240506 11:40:49.980041 18726 raft_consensus.cc:729] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842d390c25fb42329cf09feba4f2166f, State: Initialized, Role: FOLLOWER
I20240506 11:40:49.980640 18726 consensus_queue.cc:260] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 2.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "c3a9e751443445a791bcfaf771bc42b9" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 36871 } } peers { permanent_uuid: "e17b56ac16374923a53f885332de43e5" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 33737 } } peers { permanent_uuid: "842d390c25fb42329cf09feba4f2166f" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44391 } attrs { } }
I20240506 11:40:49.982046 18726 ts_tablet_manager.cc:1398] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240506 11:40:49.983152 18539 tablet_copy_service.cc:342] P e17b56ac16374923a53f885332de43e5: Request end of tablet copy session 842d390c25fb42329cf09feba4f2166f-b0da247a05c048548395d903376c4da2 received from {username='slave'} at 127.10.45.131:50111
I20240506 11:40:49.983299 18539 tablet_copy_service.cc:433] P e17b56ac16374923a53f885332de43e5: ending tablet copy session 842d390c25fb42329cf09feba4f2166f-b0da247a05c048548395d903376c4da2 on tablet b0da247a05c048548395d903376c4da2 with peer 842d390c25fb42329cf09feba4f2166f
I20240506 11:40:50.006486 18699 maintenance_manager.cc:419] P 842d390c25fb42329cf09feba4f2166f: Scheduling UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2): 550 bytes on disk
I20240506 11:40:50.007658 18589 maintenance_manager.cc:643] P 842d390c25fb42329cf09feba4f2166f: UndoDeltaBlockGCOp(b0da247a05c048548395d903376c4da2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240506 11:40:50.462505 18653 raft_consensus.cc:1203] T b0da247a05c048548395d903376c4da2 P 842d390c25fb42329cf09feba4f2166f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[2.41-2.45] Dedup: 2.45->[]
I20240506 11:40:50.474798 18744 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39326 (ReqId={client: 8b34ddf29de546aa95b02938d51487a5, seq_no=2, attempt_no=37}) took 1061 ms. Trace:
I20240506 11:40:50.474920 18744 rpcz_store.cc:276] 0506 11:40:49.413099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:49.413160 (+ 61us) service_pool.cc:225] Handling call
0506 11:40:50.474738 (+1061578us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:49.413999 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b0da247a05c048548395d903376c4da2
0506 11:40:49.414199 (+ 200us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:49.414211 (+ 12us) write_op.cc:435] Acquired schema lock
0506 11:40:49.414213 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:49.415405 (+ 1192us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:49.415474 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:49.415487 (+ 13us) tablet.cc:683] Acquiring locks for 50 operations
0506 11:40:49.416747 (+ 1260us) tablet.cc:699] Row locks acquired
0506 11:40:49.416750 (+ 3us) write_op.cc:260] PREPARE: finished
0506 11:40:49.416827 (+ 77us) write_op.cc:270] Start()
0506 11:40:49.416870 (+ 43us) write_op.cc:276] Timestamp: P: 1714995649416763 usec, L: 0
0506 11:40:49.416875 (+ 5us) op_driver.cc:331] REPLICATION: starting
0506 11:40:49.417087 (+ 212us) log.cc:842] Serialized 2455 byte log entry
0506 11:40:50.466472 (+1049385us) op_driver.cc:444] REPLICATION: finished
0506 11:40:50.471218 (+ 4746us) write_op.cc:301] APPLY: starting
0506 11:40:50.471236 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:50.472799 (+ 1563us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:50.472838 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:50.474037 (+ 1199us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:50.474192 (+ 155us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=49,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:50.474211 (+ 19us) write_op.cc:312] APPLY: finished
0506 11:40:50.474372 (+ 161us) log.cc:842] Serialized 319 byte log entry
0506 11:40:50.474497 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:50.474581 (+ 84us) write_op.cc:454] Released schema lock
0506 11:40:50.474595 (+ 14us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4673,"cfile_cache_miss":2,"cfile_cache_miss_bytes":4127,"cfile_init":1,"lbm_read_time_us":148,"lbm_reads_lt_1ms":6,"num_ops":50,"prepare.queue_time_us":410,"prepare.run_cpu_time_us":3286,"prepare.run_wall_time_us":3286,"raft.queue_time_us":125,"raft.run_cpu_time_us":872,"raft.run_wall_time_us":1092,"replication_time_us":1049552,"spinlock_wait_cycles":793856,"thread_start_us":4905,"threads_started":2}]]}
I20240506 11:40:50.477407 18744 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39326 (ReqId={client: 8b34ddf29de546aa95b02938d51487a5, seq_no=3, attempt_no=38}) took 1038 ms. Trace:
I20240506 11:40:50.477522 18744 rpcz_store.cc:276] 0506 11:40:49.439014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:49.439071 (+ 57us) service_pool.cc:225] Handling call
0506 11:40:50.477352 (+1038281us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:49.439515 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b0da247a05c048548395d903376c4da2
0506 11:40:49.439598 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:49.439603 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:40:49.439604 (+ 1us) tablet.cc:660] Decoding operations
0506 11:40:49.439854 (+ 250us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:49.439878 (+ 24us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:49.439881 (+ 3us) tablet.cc:683] Acquiring locks for 50 operations
0506 11:40:49.440070 (+ 189us) tablet.cc:699] Row locks acquired
0506 11:40:49.440071 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:40:49.440106 (+ 35us) write_op.cc:270] Start()
0506 11:40:49.440128 (+ 22us) write_op.cc:276] Timestamp: P: 1714995649440059 usec, L: 0
0506 11:40:49.440130 (+ 2us) op_driver.cc:331] REPLICATION: starting
0506 11:40:49.440288 (+ 158us) log.cc:842] Serialized 2455 byte log entry
0506 11:40:50.471172 (+1030884us) op_driver.cc:444] REPLICATION: finished
0506 11:40:50.475980 (+ 4808us) write_op.cc:301] APPLY: starting
0506 11:40:50.475989 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:50.476165 (+ 176us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:50.476184 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:50.476963 (+ 779us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:50.477016 (+ 53us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=49,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:50.477024 (+ 8us) write_op.cc:312] APPLY: finished
0506 11:40:50.477166 (+ 142us) log.cc:842] Serialized 319 byte log entry
0506 11:40:50.477238 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:50.477275 (+ 37us) write_op.cc:454] Released schema lock
0506 11:40:50.477283 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4778,"num_ops":50,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":871,"prepare.run_wall_time_us":871,"raft.queue_time_us":88,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":16,"replication_time_us":1031017,"thread_start_us":5967,"threads_started":1}]]}
I20240506 11:40:50.566901 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 18414
I20240506 11:40:50.574043 18560 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:50.657025 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 18414
I20240506 11:40:50.674858 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 18569
I20240506 11:40:50.680959 18693 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:50.763509 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 18569
I20240506 11:40:50.780601 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 18346
I20240506 11:40:50.786643 18407 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:50.877351 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 18346
2024-05-06T11:40:50Z chronyd exiting
[ OK ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy (9744 ms)
[ RUN ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL
I20240506 11:40:50.917254 10422 test_util.cc:268] Using random seed: -1970244753
I20240506 11:40:50.918988 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:40:50.919051 10422 ts_itest-base.cc:113] --------------
I20240506 11:40:50.919096 10422 ts_itest-base.cc:114] 3 tablet servers
I20240506 11:40:50.919142 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:40:50.919191 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:40:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:40:50Z Disabled control of system clock
I20240506 11:40:50.951357 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:41087
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:41087 with env {}
W20240506 11:40:51.251783 18765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:51.252068 18765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:51.252218 18765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:51.259263 18765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:40:51.259367 18765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:51.259449 18765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:40:51.259536 18765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:40:51.267948 18765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:41087
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:41087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:51.269116 18765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:51.270701 18765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:51.281193 18770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:51.282124 18773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:51.283140 18771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:51.283892 18765 server_base.cc:1034] running on GCE node
I20240506 11:40:51.284593 18765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:51.285841 18765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:51.287043 18765 hybrid_clock.cc:648] HybridClock initialized: now 1714995651286988 us; error 29 us; skew 500 ppm
I20240506 11:40:51.287364 18765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:51.289433 18765 webserver.cc:460] Webserver started at http://127.10.45.190:36067/ using document root <none> and password file <none>
I20240506 11:40:51.290072 18765 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:51.290189 18765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:51.290473 18765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:51.292876 18765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "306a7bfd4354410fac476d8b693b5a03"
format_stamp: "Formatted at 2024-05-06 11:40:51 on dist-test-slave-7l0g"
I20240506 11:40:51.293617 18765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "306a7bfd4354410fac476d8b693b5a03"
format_stamp: "Formatted at 2024-05-06 11:40:51 on dist-test-slave-7l0g"
I20240506 11:40:51.298235 18765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240506 11:40:51.301275 18779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:51.302515 18765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240506 11:40:51.302690 18765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "306a7bfd4354410fac476d8b693b5a03"
format_stamp: "Formatted at 2024-05-06 11:40:51 on dist-test-slave-7l0g"
I20240506 11:40:51.302857 18765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:51.327260 18765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:51.328243 18765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:51.328553 18765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:51.344787 18765 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:41087
I20240506 11:40:51.344805 18830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:41087 every 8 connection(s)
I20240506 11:40:51.346203 18765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:40:51.348666 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 18765
I20240506 11:40:51.348845 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:40:51.349931 18831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:51.361985 18831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03: Bootstrap starting.
I20240506 11:40:51.364569 18831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:51.365399 18831 log.cc:824] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:51.367571 18831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03: No bootstrap required, opened a new log
I20240506 11:40:51.372624 18831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a7bfd4354410fac476d8b693b5a03" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41087 } }
I20240506 11:40:51.372891 18831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:51.372987 18831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306a7bfd4354410fac476d8b693b5a03, State: Initialized, Role: FOLLOWER
I20240506 11:40:51.373560 18831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a7bfd4354410fac476d8b693b5a03" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41087 } }
I20240506 11:40:51.373752 18831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:40:51.373883 18831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:40:51.374023 18831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:51.375833 18831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a7bfd4354410fac476d8b693b5a03" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41087 } }
I20240506 11:40:51.376211 18831 leader_election.cc:302] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 306a7bfd4354410fac476d8b693b5a03; no voters:
I20240506 11:40:51.376772 18831 leader_election.cc:288] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:40:51.376902 18836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:51.377529 18836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [term 1 LEADER]: Becoming Leader. State: Replica: 306a7bfd4354410fac476d8b693b5a03, State: Running, Role: LEADER
I20240506 11:40:51.377991 18836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a7bfd4354410fac476d8b693b5a03" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41087 } }
I20240506 11:40:51.378451 18831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:40:51.380481 18837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "306a7bfd4354410fac476d8b693b5a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a7bfd4354410fac476d8b693b5a03" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41087 } } }
I20240506 11:40:51.380815 18837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:51.380460 18838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 306a7bfd4354410fac476d8b693b5a03. Latest consensus state: current_term: 1 leader_uuid: "306a7bfd4354410fac476d8b693b5a03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306a7bfd4354410fac476d8b693b5a03" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 41087 } } }
I20240506 11:40:51.381359 18838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03 [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:51.382917 18842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:40:51.387570 18842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:40:51.395251 18842 catalog_manager.cc:1347] Generated new cluster ID: c34711fc6f45470bb335d6f5c2bc9a2d
I20240506 11:40:51.395359 18842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:40:51.415431 18842 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:40:51.416872 18842 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:40:51.431041 18842 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 306a7bfd4354410fac476d8b693b5a03: Generated new TSK 0
I20240506 11:40:51.431903 18842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:40:51.437259 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41087
--builtin_ntp_servers=127.10.45.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
W20240506 11:40:51.734241 18855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:51.734503 18855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:51.734649 18855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:51.741492 18855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:51.741676 18855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:40:51.749630 18855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:51.750690 18855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:51.752163 18855 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:51.762864 18861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:51.763712 18863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:51.764390 18860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:51.765367 18855 server_base.cc:1034] running on GCE node
I20240506 11:40:51.765908 18855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:51.766695 18855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:51.767894 18855 hybrid_clock.cc:648] HybridClock initialized: now 1714995651767846 us; error 25 us; skew 500 ppm
I20240506 11:40:51.768111 18855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:51.769937 18855 webserver.cc:460] Webserver started at http://127.10.45.129:46715/ using document root <none> and password file <none>
I20240506 11:40:51.770496 18855 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:51.770588 18855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:51.770805 18855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:51.774549 18855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d7ae310d7bd1414bb1fe9c0400150a3c"
format_stamp: "Formatted at 2024-05-06 11:40:51 on dist-test-slave-7l0g"
I20240506 11:40:51.775249 18855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d7ae310d7bd1414bb1fe9c0400150a3c"
format_stamp: "Formatted at 2024-05-06 11:40:51 on dist-test-slave-7l0g"
I20240506 11:40:51.779450 18855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:40:51.782284 18869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:51.783445 18855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:51.783571 18855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d7ae310d7bd1414bb1fe9c0400150a3c"
format_stamp: "Formatted at 2024-05-06 11:40:51 on dist-test-slave-7l0g"
I20240506 11:40:51.783726 18855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:51.802275 18855 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:51.803023 18855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:51.803325 18855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:51.804080 18855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:51.805274 18855 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:51.805341 18855 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:51.805435 18855 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:51.805480 18855 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:51.832839 18855 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:39925
I20240506 11:40:51.832878 18981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:39925 every 8 connection(s)
I20240506 11:40:51.834005 18855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:40:51.837158 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 18855
I20240506 11:40:51.837368 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:40:51.840489 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41087
--builtin_ntp_servers=127.10.45.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20240506 11:40:51.847167 18982 heartbeater.cc:347] Connected to a master server at 127.10.45.190:41087
I20240506 11:40:51.847482 18982 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:51.848457 18982 heartbeater.cc:511] Master 127.10.45.190:41087 requested a full tablet report, sending...
I20240506 11:40:51.850756 18796 ts_manager.cc:194] Registered new tserver with Master: d7ae310d7bd1414bb1fe9c0400150a3c (127.10.45.129:39925)
I20240506 11:40:51.852892 18796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:60819
W20240506 11:40:52.142781 18985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:52.143074 18985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:52.143188 18985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:52.149737 18985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:52.149883 18985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:40:52.157605 18985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:52.158653 18985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:52.160246 18985 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:52.171474 18991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:52.172295 18993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:52.173112 18990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:52.174037 18985 server_base.cc:1034] running on GCE node
I20240506 11:40:52.174620 18985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:52.175487 18985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:52.176760 18985 hybrid_clock.cc:648] HybridClock initialized: now 1714995652176686 us; error 37 us; skew 500 ppm
I20240506 11:40:52.177178 18985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:52.179493 18985 webserver.cc:460] Webserver started at http://127.10.45.130:42611/ using document root <none> and password file <none>
I20240506 11:40:52.180177 18985 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:52.180308 18985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:52.180580 18985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:52.183071 18985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "af8e6e8fea9c4676aafc4833d2c1010e"
format_stamp: "Formatted at 2024-05-06 11:40:52 on dist-test-slave-7l0g"
I20240506 11:40:52.183825 18985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "af8e6e8fea9c4676aafc4833d2c1010e"
format_stamp: "Formatted at 2024-05-06 11:40:52 on dist-test-slave-7l0g"
I20240506 11:40:52.188541 18985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240506 11:40:52.191556 18999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:52.192755 18985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:52.192905 18985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "af8e6e8fea9c4676aafc4833d2c1010e"
format_stamp: "Formatted at 2024-05-06 11:40:52 on dist-test-slave-7l0g"
I20240506 11:40:52.193060 18985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:52.220670 18985 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:52.221467 18985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:52.221751 18985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:52.222669 18985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:52.224081 18985 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:52.224162 18985 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:52.224292 18985 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:52.224347 18985 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:52.253365 18985 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:32815
I20240506 11:40:52.253424 19111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:32815 every 8 connection(s)
I20240506 11:40:52.254489 18985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:40:52.255270 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 18985
I20240506 11:40:52.255446 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:40:52.258502 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41087
--builtin_ntp_servers=127.10.45.148:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20240506 11:40:52.266752 19112 heartbeater.cc:347] Connected to a master server at 127.10.45.190:41087
I20240506 11:40:52.267040 19112 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:52.267786 19112 heartbeater.cc:511] Master 127.10.45.190:41087 requested a full tablet report, sending...
I20240506 11:40:52.269101 18796 ts_manager.cc:194] Registered new tserver with Master: af8e6e8fea9c4676aafc4833d2c1010e (127.10.45.130:32815)
I20240506 11:40:52.269768 18796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:53651
W20240506 11:40:52.573832 19115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:52.574118 19115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:52.574265 19115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:52.581442 19115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:52.581645 19115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:40:52.590480 19115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:41087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:52.591785 19115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:52.593538 19115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:52.605796 19120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:52.606724 19123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:52.607815 19121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:52.608609 19115 server_base.cc:1034] running on GCE node
I20240506 11:40:52.609215 19115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:52.610162 19115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:52.611364 19115 hybrid_clock.cc:648] HybridClock initialized: now 1714995652611290 us; error 52 us; skew 500 ppm
I20240506 11:40:52.611639 19115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:52.613770 19115 webserver.cc:460] Webserver started at http://127.10.45.131:45795/ using document root <none> and password file <none>
I20240506 11:40:52.614434 19115 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:52.614537 19115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:52.614790 19115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:52.617280 19115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "db45f34c8cf647f5ad920ba7c9786cc6"
format_stamp: "Formatted at 2024-05-06 11:40:52 on dist-test-slave-7l0g"
I20240506 11:40:52.618073 19115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "db45f34c8cf647f5ad920ba7c9786cc6"
format_stamp: "Formatted at 2024-05-06 11:40:52 on dist-test-slave-7l0g"
I20240506 11:40:52.623310 19115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240506 11:40:52.626600 19129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:52.627961 19115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240506 11:40:52.628131 19115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "db45f34c8cf647f5ad920ba7c9786cc6"
format_stamp: "Formatted at 2024-05-06 11:40:52 on dist-test-slave-7l0g"
I20240506 11:40:52.628314 19115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:52.656231 19115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:52.657012 19115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:52.657298 19115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:52.658193 19115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:52.659787 19115 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:52.659871 19115 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:52.659999 19115 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:52.660077 19115 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:52.690589 19115 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:43999
I20240506 11:40:52.690609 19241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:43999 every 8 connection(s)
I20240506 11:40:52.691882 19115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:40:52.700892 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 19115
I20240506 11:40:52.701072 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:40:52.703732 19242 heartbeater.cc:347] Connected to a master server at 127.10.45.190:41087
I20240506 11:40:52.704037 19242 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:52.704800 19242 heartbeater.cc:511] Master 127.10.45.190:41087 requested a full tablet report, sending...
I20240506 11:40:52.705960 18796 ts_manager.cc:194] Registered new tserver with Master: db45f34c8cf647f5ad920ba7c9786cc6 (127.10.45.131:43999)
I20240506 11:40:52.706705 18796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:53637
I20240506 11:40:52.716096 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:40:52.730191 18796 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54222:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20240506 11:40:52.735603 18796 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 11:40:52.762076 19047 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad66490be8f488d9fb420db6bb128b4 (DEFAULT_TABLE table=TestTable [id=40989a9275e242f8bcb53b7fecbb9f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:52.761970 18917 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad66490be8f488d9fb420db6bb128b4 (DEFAULT_TABLE table=TestTable [id=40989a9275e242f8bcb53b7fecbb9f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:52.762461 19177 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad66490be8f488d9fb420db6bb128b4 (DEFAULT_TABLE table=TestTable [id=40989a9275e242f8bcb53b7fecbb9f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 11:40:52.763325 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad66490be8f488d9fb420db6bb128b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:52.763402 18917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad66490be8f488d9fb420db6bb128b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:52.763727 19177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad66490be8f488d9fb420db6bb128b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:52.773408 19260 tablet_bootstrap.cc:492] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e: Bootstrap starting.
I20240506 11:40:52.775269 19261 tablet_bootstrap.cc:492] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: Bootstrap starting.
I20240506 11:40:52.776010 19262 tablet_bootstrap.cc:492] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6: Bootstrap starting.
I20240506 11:40:52.776660 19260 tablet_bootstrap.cc:654] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:52.777873 19260 log.cc:824] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:52.778342 19261 tablet_bootstrap.cc:654] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:52.779086 19262 tablet_bootstrap.cc:654] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:52.779520 19261 log.cc:824] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:52.780230 19262 log.cc:824] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:52.781183 19260 tablet_bootstrap.cc:492] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e: No bootstrap required, opened a new log
I20240506 11:40:52.781503 19260 ts_tablet_manager.cc:1367] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240506 11:40:52.781983 19261 tablet_bootstrap.cc:492] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: No bootstrap required, opened a new log
I20240506 11:40:52.782212 19261 ts_tablet_manager.cc:1367] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240506 11:40:52.782815 19262 tablet_bootstrap.cc:492] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6: No bootstrap required, opened a new log
I20240506 11:40:52.783071 19262 ts_tablet_manager.cc:1367] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20240506 11:40:52.787629 19261 raft_consensus.cc:348] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 } } peers { permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 } } peers { permanent_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39925 } }
I20240506 11:40:52.788129 19261 raft_consensus.cc:374] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:52.788237 19261 raft_consensus.cc:729] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ae310d7bd1414bb1fe9c0400150a3c, State: Initialized, Role: FOLLOWER
I20240506 11:40:52.788867 19261 consensus_queue.cc:260] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 } } peers { permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 } } peers { permanent_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39925 } }
I20240506 11:40:52.789884 18982 heartbeater.cc:503] Master 127.10.45.190:41087 was elected leader, sending a full tablet report...
I20240506 11:40:52.790506 19261 ts_tablet_manager.cc:1398] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240506 11:40:52.790284 19260 raft_consensus.cc:348] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 } } peers { permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 } } peers { permanent_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39925 } }
I20240506 11:40:52.790645 19260 raft_consensus.cc:374] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:52.790742 19260 raft_consensus.cc:729] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8e6e8fea9c4676aafc4833d2c1010e, State: Initialized, Role: FOLLOWER
I20240506 11:40:52.791090 19262 raft_consensus.cc:348] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 } } peers { permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 } } peers { permanent_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39925 } }
I20240506 11:40:52.791465 19262 raft_consensus.cc:374] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:52.791378 19260 consensus_queue.cc:260] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 } } peers { permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 } } peers { permanent_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39925 } }
I20240506 11:40:52.791556 19262 raft_consensus.cc:729] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db45f34c8cf647f5ad920ba7c9786cc6, State: Initialized, Role: FOLLOWER
I20240506 11:40:52.792392 19112 heartbeater.cc:503] Master 127.10.45.190:41087 was elected leader, sending a full tablet report...
I20240506 11:40:52.792881 19260 ts_tablet_manager.cc:1398] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240506 11:40:52.792994 19262 consensus_queue.cc:260] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 } } peers { permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 } } peers { permanent_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39925 } }
I20240506 11:40:52.794006 19242 heartbeater.cc:503] Master 127.10.45.190:41087 was elected leader, sending a full tablet report...
I20240506 11:40:52.794553 19262 ts_tablet_manager.cc:1398] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20240506 11:40:52.795403 19266 raft_consensus.cc:482] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 11:40:52.795581 19266 raft_consensus.cc:504] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 } } peers { permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 } } peers { permanent_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39925 } }
I20240506 11:40:52.797595 19266 leader_election.cc:288] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db45f34c8cf647f5ad920ba7c9786cc6 (127.10.45.131:43999), af8e6e8fea9c4676aafc4833d2c1010e (127.10.45.130:32815)
I20240506 11:40:52.805081 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" is_pre_election: true
I20240506 11:40:52.805228 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: true
I20240506 11:40:52.805593 19197 raft_consensus.cc:2454] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ae310d7bd1414bb1fe9c0400150a3c in term 0.
I20240506 11:40:52.805721 19067 raft_consensus.cc:2454] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ae310d7bd1414bb1fe9c0400150a3c in term 0.
I20240506 11:40:52.806248 18873 leader_election.cc:302] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7ae310d7bd1414bb1fe9c0400150a3c, db45f34c8cf647f5ad920ba7c9786cc6; no voters:
I20240506 11:40:52.806660 19266 raft_consensus.cc:2781] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 11:40:52.806783 19266 raft_consensus.cc:482] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 11:40:52.806864 19266 raft_consensus.cc:3037] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:52.808760 19266 raft_consensus.cc:504] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 } } peers { permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 } } peers { permanent_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39925 } }
I20240506 11:40:52.809270 19266 leader_election.cc:288] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [CANDIDATE]: Term 1 election: Requested vote from peers db45f34c8cf647f5ad920ba7c9786cc6 (127.10.45.131:43999), af8e6e8fea9c4676aafc4833d2c1010e (127.10.45.130:32815)
I20240506 11:40:52.809749 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db45f34c8cf647f5ad920ba7c9786cc6"
I20240506 11:40:52.809823 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e"
I20240506 11:40:52.809924 19197 raft_consensus.cc:3037] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:52.809984 19067 raft_consensus.cc:3037] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:52.812294 19067 raft_consensus.cc:2454] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ae310d7bd1414bb1fe9c0400150a3c in term 1.
I20240506 11:40:52.812294 19197 raft_consensus.cc:2454] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ae310d7bd1414bb1fe9c0400150a3c in term 1.
I20240506 11:40:52.812676 18873 leader_election.cc:302] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af8e6e8fea9c4676aafc4833d2c1010e, d7ae310d7bd1414bb1fe9c0400150a3c; no voters:
I20240506 11:40:52.812966 19266 raft_consensus.cc:2781] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:52.813334 19266 raft_consensus.cc:686] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [term 1 LEADER]: Becoming Leader. State: Replica: d7ae310d7bd1414bb1fe9c0400150a3c, State: Running, Role: LEADER
I20240506 11:40:52.813774 19266 consensus_queue.cc:237] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 } } peers { permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 } } peers { permanent_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39925 } }
I20240506 11:40:52.817883 18795 catalog_manager.cc:5484] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c reported cstate change: term changed from 0 to 1, leader changed from <none> to d7ae310d7bd1414bb1fe9c0400150a3c (127.10.45.129). New cstate: current_term: 1 leader_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 39925 } health_report { overall_health: HEALTHY } } }
I20240506 11:40:52.832799 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:40:52.834929 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d7ae310d7bd1414bb1fe9c0400150a3c to finish bootstrapping
W20240506 11:40:52.835981 18983 tablet.cc:2369] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:40:52.843479 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver af8e6e8fea9c4676aafc4833d2c1010e to finish bootstrapping
I20240506 11:40:52.850558 10422 ts_itest-base.cc:233] Waiting for 1 tablets on tserver db45f34c8cf647f5ad920ba7c9786cc6 to finish bootstrapping
I20240506 11:40:52.858232 18977 generic_service.cc:175] {username='slave'} at 127.0.0.1:60988 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20240506 11:40:52.859716 19107 generic_service.cc:175] {username='slave'} at 127.0.0.1:59136 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20240506 11:40:52.861155 19237 generic_service.cc:175] {username='slave'} at 127.0.0.1:44180 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20240506 11:40:52.876843 19197 raft_consensus.cc:1261] T 5ad66490be8f488d9fb420db6bb128b4 P db45f34c8cf647f5ad920ba7c9786cc6 [term 1 FOLLOWER]: Refusing update from remote peer d7ae310d7bd1414bb1fe9c0400150a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:52.876842 19067 raft_consensus.cc:1261] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e [term 1 FOLLOWER]: Refusing update from remote peer d7ae310d7bd1414bb1fe9c0400150a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240506 11:40:52.877667 19271 consensus_queue.cc:1035] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:52.877948 19266 consensus_queue.cc:1035] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 11:40:52.888434 19275 mvcc.cc:205] Tried to move back new op lower bound from 7024622194174803968 to 7024622193928806400. Current Snapshot: MvccSnapshot[applied={T|T < 7024622194174803968}]
I20240506 11:40:52.907858 19107 generic_service.cc:175] {username='slave'} at 127.0.0.1:59136 changed flags via RPC: log_inject_latency_ms_mean from '100' to '600'
I20240506 11:40:52.908843 19107 generic_service.cc:175] {username='slave'} at 127.0.0.1:59136 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20240506 11:40:52.909732 19107 generic_service.cc:175] {username='slave'} at 127.0.0.1:59136 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20240506 11:40:52.910682 18977 generic_service.cc:175] {username='slave'} at 127.0.0.1:60988 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1200'
I20240506 11:40:52.911589 18977 generic_service.cc:175] {username='slave'} at 127.0.0.1:60988 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20240506 11:40:52.912294 18977 generic_service.cc:175] {username='slave'} at 127.0.0.1:60988 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20240506 11:40:52.912631 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 19115
I20240506 11:40:52.913347 19237 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:52.994349 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 19115
I20240506 11:40:53.014351 10422 test_util.cc:268] Using random seed: -1968147673
I20240506 11:40:53.032061 10422 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20240506 11:40:53.043999 19272 log.cc:508] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20240506 11:40:53.044596 18873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.45.131:43999: connect: Connection refused (error 111)
W20240506 11:40:53.046046 18873 consensus_peers.cc:485] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c -> Peer db45f34c8cf647f5ad920ba7c9786cc6 (127.10.45.131:43999): Couldn't send request to peer db45f34c8cf647f5ad920ba7c9786cc6. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.131:43999: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240506 11:40:53.046203 19279 log.cc:508] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e: Injecting 600ms of latency in SegmentAllocator::Sync()
I20240506 11:40:53.083799 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.086889 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.139482 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.141781 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.194305 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.196694 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.249265 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.251366 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.303941 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.305979 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.358525 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.360721 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.413165 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.415436 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.468005 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.470183 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.522615 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.524736 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.577142 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.579427 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.631973 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.634335 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.686861 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.689184 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.741583 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.743736 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.796159 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.798430 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.850936 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.853140 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.905555 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.908094 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:53.960726 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:53.963104 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.015635 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.017848 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
W20240506 11:40:54.067322 18873 consensus_peers.cc:485] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c -> Peer db45f34c8cf647f5ad920ba7c9786cc6 (127.10.45.131:43999): Couldn't send request to peer db45f34c8cf647f5ad920ba7c9786cc6. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.131:43999: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240506 11:40:54.070183 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.072149 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.124465 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.126796 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.179116 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.181210 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.233659 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.235760 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.245728 19293 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32772 (ReqId={client: 263e746af55b40e789671b53fd1b7525, seq_no=0, attempt_no=0}) took 1204 ms. Trace:
I20240506 11:40:54.245849 19293 rpcz_store.cc:276] 0506 11:40:53.041727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:53.041795 (+ 68us) service_pool.cc:225] Handling call
0506 11:40:54.245677 (+1203882us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:53.042854 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 5ad66490be8f488d9fb420db6bb128b4
0506 11:40:53.042968 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:53.042977 (+ 9us) write_op.cc:435] Acquired schema lock
0506 11:40:53.042980 (+ 3us) tablet.cc:660] Decoding operations
0506 11:40:53.043058 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:53.043069 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:53.043072 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:53.043113 (+ 41us) tablet.cc:699] Row locks acquired
0506 11:40:53.043115 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:53.043257 (+ 142us) write_op.cc:270] Start()
0506 11:40:53.043301 (+ 44us) write_op.cc:276] Timestamp: P: 1714995653043223 usec, L: 0
0506 11:40:53.043304 (+ 3us) op_driver.cc:331] REPLICATION: starting
0506 11:40:53.043524 (+ 220us) log.cc:842] Serialized 345 byte log entry
0506 11:40:54.244501 (+1200977us) op_driver.cc:444] REPLICATION: finished
0506 11:40:54.244883 (+ 382us) write_op.cc:301] APPLY: starting
0506 11:40:54.244903 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:54.244939 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:54.244962 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:54.245054 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:54.245087 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:54.245092 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:40:54.245169 (+ 77us) log.cc:842] Serialized 22 byte log entry
0506 11:40:54.245297 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:54.245320 (+ 23us) write_op.cc:454] Released schema lock
0506 11:40:54.245327 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":298,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":792,"prepare.run_wall_time_us":791,"raft.queue_time_us":127,"raft.run_cpu_time_us":672,"raft.run_wall_time_us":669,"replication_time_us":1201172,"spinlock_wait_cycles":46976,"thread_start_us":209,"threads_started":1}]]}
I20240506 11:40:54.287843 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.289796 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.342078 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.344435 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.396847 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.399214 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
W20240506 11:40:54.450210 19272 log.cc:508] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: Injecting 1200ms of latency in SegmentAllocator::Sync()
I20240506 11:40:54.451433 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
W20240506 11:40:54.452085 19279 log.cc:508] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e: Injecting 600ms of latency in SegmentAllocator::Sync()
I20240506 11:40:54.453655 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.506011 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.508376 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.560688 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.563074 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.615450 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.617551 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.669741 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.671952 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.724319 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.727025 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.779323 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.781473 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.833825 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.836205 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.888291 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.890350 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.942662 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.944752 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:54.996948 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:54.999161 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
W20240506 11:40:55.023706 18873 consensus_peers.cc:485] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c -> Peer db45f34c8cf647f5ad920ba7c9786cc6 (127.10.45.131:43999): Couldn't send request to peer db45f34c8cf647f5ad920ba7c9786cc6. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.131:43999: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240506 11:40:55.051605 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.054046 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.106328 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.108436 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.160809 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.162936 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.215214 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.217274 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.269608 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.271697 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.323976 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.325912 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.378054 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.380364 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.432778 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.434834 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.487197 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.489348 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.541582 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.543680 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.595976 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.598055 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.650628 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.652180 19299 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32772 (ReqId={client: 263e746af55b40e789671b53fd1b7525, seq_no=1, attempt_no=0}) took 1203 ms. Trace:
I20240506 11:40:55.652397 19299 rpcz_store.cc:276] 0506 11:40:54.448359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:54.448445 (+ 86us) service_pool.cc:225] Handling call
0506 11:40:55.652097 (+1203652us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:54.449376 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 5ad66490be8f488d9fb420db6bb128b4
0506 11:40:54.449485 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:54.449490 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:40:54.449492 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:54.449572 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:54.449583 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:54.449586 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:54.449639 (+ 53us) tablet.cc:699] Row locks acquired
0506 11:40:54.449641 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:54.449678 (+ 37us) write_op.cc:270] Start()
0506 11:40:54.449708 (+ 30us) write_op.cc:276] Timestamp: P: 1714995654449657 usec, L: 0
0506 11:40:54.449710 (+ 2us) op_driver.cc:331] REPLICATION: starting
0506 11:40:54.449874 (+ 164us) log.cc:842] Serialized 354 byte log entry
0506 11:40:55.650723 (+1200849us) op_driver.cc:444] REPLICATION: finished
0506 11:40:55.651290 (+ 567us) write_op.cc:301] APPLY: starting
0506 11:40:55.651319 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:55.651407 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:55.651437 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:55.651528 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:55.651602 (+ 74us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:55.651614 (+ 12us) write_op.cc:312] APPLY: finished
0506 11:40:55.651758 (+ 144us) log.cc:842] Serialized 22 byte log entry
0506 11:40:55.651904 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:55.651946 (+ 42us) write_op.cc:454] Released schema lock
0506 11:40:55.651957 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":480,"num_ops":1,"prepare.queue_time_us":348,"prepare.run_cpu_time_us":620,"prepare.run_wall_time_us":620,"raft.queue_time_us":100,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":263,"replication_time_us":1200989,"thread_start_us":595,"threads_started":2}]]}
I20240506 11:40:55.654481 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.706812 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.709081 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.761297 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.763511 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.815917 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.818058 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
W20240506 11:40:55.856418 19272 log.cc:508] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20240506 11:40:55.857931 19279 log.cc:508] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e: Injecting 600ms of latency in SegmentAllocator::Sync()
I20240506 11:40:55.870419 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.872608 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:55.924945 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.927274 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
W20240506 11:40:55.973529 18873 consensus_peers.cc:485] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c -> Peer db45f34c8cf647f5ad920ba7c9786cc6 (127.10.45.131:43999): Couldn't send request to peer db45f34c8cf647f5ad920ba7c9786cc6. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.131:43999: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240506 11:40:55.979593 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:55.981676 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.034808 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.036900 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.089177 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.091444 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.143754 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.145795 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.198037 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.200129 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.252599 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.254735 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.307286 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.309511 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.361771 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.363818 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.416014 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.418408 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.470856 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.473039 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.525342 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.527529 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.579916 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.582070 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.634449 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.636684 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.689056 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.691268 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.743633 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.745756 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.798013 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.800163 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.852357 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.855239 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.908003 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.910003 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:56.962319 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:56.964604 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
W20240506 11:40:57.014084 18873 consensus_peers.cc:485] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c -> Peer db45f34c8cf647f5ad920ba7c9786cc6 (127.10.45.131:43999): Couldn't send request to peer db45f34c8cf647f5ad920ba7c9786cc6. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.131:43999: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240506 11:40:57.016845 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.018721 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.057897 19305 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32772 (ReqId={client: 263e746af55b40e789671b53fd1b7525, seq_no=2, attempt_no=0}) took 1203 ms. Trace:
I20240506 11:40:57.058348 19305 rpcz_store.cc:276] 0506 11:40:55.854717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:55.854811 (+ 94us) service_pool.cc:225] Handling call
0506 11:40:57.057822 (+1203011us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:55.855661 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 5ad66490be8f488d9fb420db6bb128b4
0506 11:40:55.855757 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:55.855761 (+ 4us) write_op.cc:435] Acquired schema lock
0506 11:40:55.855763 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:55.855840 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:55.855849 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:55.855852 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:55.855894 (+ 42us) tablet.cc:699] Row locks acquired
0506 11:40:55.855895 (+ 1us) write_op.cc:260] PREPARE: finished
0506 11:40:55.855925 (+ 30us) write_op.cc:270] Start()
0506 11:40:55.855947 (+ 22us) write_op.cc:276] Timestamp: P: 1714995655855886 usec, L: 0
0506 11:40:55.855949 (+ 2us) op_driver.cc:331] REPLICATION: starting
0506 11:40:55.856104 (+ 155us) log.cc:842] Serialized 354 byte log entry
0506 11:40:57.056929 (+1200825us) op_driver.cc:444] REPLICATION: finished
0506 11:40:57.057301 (+ 372us) write_op.cc:301] APPLY: starting
0506 11:40:57.057317 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:57.057361 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:57.057399 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:57.057472 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:57.057503 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:57.057507 (+ 4us) write_op.cc:312] APPLY: finished
0506 11:40:57.057579 (+ 72us) log.cc:842] Serialized 22 byte log entry
0506 11:40:57.057695 (+ 116us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:57.057715 (+ 20us) write_op.cc:454] Released schema lock
0506 11:40:57.057722 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":316,"num_ops":1,"prepare.queue_time_us":335,"prepare.run_cpu_time_us":569,"prepare.run_wall_time_us":569,"raft.queue_time_us":124,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":229,"replication_time_us":1200954,"thread_start_us":478,"threads_started":2}]]}
I20240506 11:40:57.070852 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.072717 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.125041 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.127024 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.179279 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.181332 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.233325 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.235181 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
W20240506 11:40:57.261642 19272 log.cc:508] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20240506 11:40:57.263274 19279 log.cc:508] T 5ad66490be8f488d9fb420db6bb128b4 P af8e6e8fea9c4676aafc4833d2c1010e: Injecting 600ms of latency in SegmentAllocator::Sync()
I20240506 11:40:57.287170 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.288960 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.341074 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.343221 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.395378 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.397420 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.449674 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.451807 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.504046 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.506078 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.558468 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.560570 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.612766 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.614871 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.667135 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.669061 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.721349 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.723407 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.775676 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.777714 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.830176 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.832234 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.884781 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.886763 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.939231 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.941392 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:57.993626 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:57.995832 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:58.048329 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:58.050487 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
W20240506 11:40:58.070876 18873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.45.131:43999: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240506 11:40:58.071599 18873 consensus_peers.cc:485] T 5ad66490be8f488d9fb420db6bb128b4 P d7ae310d7bd1414bb1fe9c0400150a3c -> Peer db45f34c8cf647f5ad920ba7c9786cc6 (127.10.45.131:43999): Couldn't send request to peer db45f34c8cf647f5ad920ba7c9786cc6. Status: Network error: Client connection negotiation failed: client connection to 127.10.45.131:43999: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240506 11:40:58.102833 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:58.105067 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:58.157359 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:58.159552 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:58.211824 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:58.213982 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:58.266332 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:58.268393 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:58.320744 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:58.322839 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:58.375291 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:58.377583 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:58.429883 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:58.431934 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:58.463449 19311 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32772 (ReqId={client: 263e746af55b40e789671b53fd1b7525, seq_no=3, attempt_no=0}) took 1203 ms. Trace:
I20240506 11:40:58.463614 19311 rpcz_store.cc:276] 0506 11:40:57.259961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0506 11:40:57.260062 (+ 101us) service_pool.cc:225] Handling call
0506 11:40:58.463367 (+1203305us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0506 11:40:57.260869 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 5ad66490be8f488d9fb420db6bb128b4
0506 11:40:57.260970 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0506 11:40:57.260975 (+ 5us) write_op.cc:435] Acquired schema lock
0506 11:40:57.260977 (+ 2us) tablet.cc:660] Decoding operations
0506 11:40:57.261052 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0506 11:40:57.261063 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0506 11:40:57.261065 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0506 11:40:57.261113 (+ 48us) tablet.cc:699] Row locks acquired
0506 11:40:57.261115 (+ 2us) write_op.cc:260] PREPARE: finished
0506 11:40:57.261146 (+ 31us) write_op.cc:270] Start()
0506 11:40:57.261171 (+ 25us) write_op.cc:276] Timestamp: P: 1714995657261109 usec, L: 0
0506 11:40:57.261173 (+ 2us) op_driver.cc:331] REPLICATION: starting
0506 11:40:57.261324 (+ 151us) log.cc:842] Serialized 354 byte log entry
0506 11:40:58.462198 (+1200874us) op_driver.cc:444] REPLICATION: finished
0506 11:40:58.462802 (+ 604us) write_op.cc:301] APPLY: starting
0506 11:40:58.462823 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0506 11:40:58.462864 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0506 11:40:58.462894 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0506 11:40:58.462994 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0506 11:40:58.463023 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0506 11:40:58.463028 (+ 5us) write_op.cc:312] APPLY: finished
0506 11:40:58.463108 (+ 80us) log.cc:842] Serialized 22 byte log entry
0506 11:40:58.463228 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0506 11:40:58.463250 (+ 22us) write_op.cc:454] Released schema lock
0506 11:40:58.463258 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":494,"num_ops":1,"prepare.queue_time_us":334,"prepare.run_cpu_time_us":596,"prepare.run_wall_time_us":596,"raft.queue_time_us":128,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":232,"replication_time_us":1200996,"thread_start_us":635,"threads_started":2}]]}
I20240506 11:40:58.484270 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "d7ae310d7bd1414bb1fe9c0400150a3c" is_pre_election: false
I20240506 11:40:58.486388 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad66490be8f488d9fb420db6bb128b4" candidate_uuid: "db45f34c8cf647f5ad920ba7c9786cc6" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "af8e6e8fea9c4676aafc4833d2c1010e" is_pre_election: false
I20240506 11:40:58.667223 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 18855
I20240506 11:40:58.668116 18977 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:58.751950 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 18855
I20240506 11:40:58.768522 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 18985
I20240506 11:40:58.769384 19107 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:58.852311 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 18985
I20240506 11:40:58.869364 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 18765
I20240506 11:40:58.870430 18826 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:40:58.967742 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 18765
2024-05-06T11:40:58Z chronyd exiting
[ OK ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL (8089 ms)
[----------] 12 tests from RaftConsensusElectionITest (184451 ms total)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0
I20240506 11:40:59.006407 10422 test_util.cc:268] Using random seed: -1962155617
I20240506 11:40:59.008139 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:40:59.008212 10422 ts_itest-base.cc:113] --------------
I20240506 11:40:59.008250 10422 ts_itest-base.cc:114] 1 tablet servers
I20240506 11:40:59.008287 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:40:59.008318 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:40:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:40:59Z Disabled control of system clock
I20240506 11:40:59.043641 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:45763
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:45763
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20240506 11:40:59.350631 19325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:59.350904 19325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:59.351058 19325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:59.359205 19325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:40:59.359313 19325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:59.359354 19325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:40:59.359411 19325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:40:59.368031 19325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:45763
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:59.369266 19325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:59.370864 19325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:59.381542 19330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:59.382401 19331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:59.383195 19333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:59.383960 19325 server_base.cc:1034] running on GCE node
I20240506 11:40:59.384685 19325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:59.385926 19325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:59.387128 19325 hybrid_clock.cc:648] HybridClock initialized: now 1714995659387072 us; error 34 us; skew 500 ppm
I20240506 11:40:59.387406 19325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:59.389413 19325 webserver.cc:460] Webserver started at http://127.10.45.190:42461/ using document root <none> and password file <none>
I20240506 11:40:59.390058 19325 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:59.390177 19325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:59.390532 19325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:59.393062 19325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c9d18e1ec1d74d378dd4a7c374641ebe"
format_stamp: "Formatted at 2024-05-06 11:40:59 on dist-test-slave-7l0g"
I20240506 11:40:59.393723 19325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c9d18e1ec1d74d378dd4a7c374641ebe"
format_stamp: "Formatted at 2024-05-06 11:40:59 on dist-test-slave-7l0g"
I20240506 11:40:59.398180 19325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240506 11:40:59.401140 19339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:59.402283 19325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 11:40:59.402422 19325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c9d18e1ec1d74d378dd4a7c374641ebe"
format_stamp: "Formatted at 2024-05-06 11:40:59 on dist-test-slave-7l0g"
I20240506 11:40:59.402562 19325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:59.420609 19325 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:59.421387 19325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:59.421654 19325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:59.437552 19325 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:45763
I20240506 11:40:59.437580 19390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:45763 every 8 connection(s)
I20240506 11:40:59.438755 19325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:40:59.442281 19391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:59.443399 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 19325
I20240506 11:40:59.443626 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:40:59.452351 19391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe: Bootstrap starting.
I20240506 11:40:59.455749 19391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe: Neither blocks nor log segments found. Creating new log.
I20240506 11:40:59.456707 19391 log.cc:824] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe: Log is configured to *not* fsync() on all Append() calls
I20240506 11:40:59.459004 19391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe: No bootstrap required, opened a new log
I20240506 11:40:59.464663 19391 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d18e1ec1d74d378dd4a7c374641ebe" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45763 } }
I20240506 11:40:59.464947 19391 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:40:59.465032 19391 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d18e1ec1d74d378dd4a7c374641ebe, State: Initialized, Role: FOLLOWER
I20240506 11:40:59.465550 19391 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d18e1ec1d74d378dd4a7c374641ebe" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45763 } }
I20240506 11:40:59.465737 19391 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:40:59.465823 19391 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:40:59.465934 19391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:40:59.468010 19391 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d18e1ec1d74d378dd4a7c374641ebe" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45763 } }
I20240506 11:40:59.468408 19391 leader_election.cc:302] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d18e1ec1d74d378dd4a7c374641ebe; no voters:
I20240506 11:40:59.468950 19391 leader_election.cc:288] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:40:59.469075 19396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:40:59.469653 19396 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [term 1 LEADER]: Becoming Leader. State: Replica: c9d18e1ec1d74d378dd4a7c374641ebe, State: Running, Role: LEADER
I20240506 11:40:59.470175 19396 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d18e1ec1d74d378dd4a7c374641ebe" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45763 } }
I20240506 11:40:59.470754 19391 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:40:59.472682 19397 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9d18e1ec1d74d378dd4a7c374641ebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d18e1ec1d74d378dd4a7c374641ebe" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45763 } } }
I20240506 11:40:59.472923 19397 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:59.474418 19402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:40:59.474388 19398 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d18e1ec1d74d378dd4a7c374641ebe. Latest consensus state: current_term: 1 leader_uuid: "c9d18e1ec1d74d378dd4a7c374641ebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d18e1ec1d74d378dd4a7c374641ebe" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 45763 } } }
I20240506 11:40:59.474625 19398 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe [sys.catalog]: This master's current role is: LEADER
I20240506 11:40:59.478379 19402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:40:59.486934 19402 catalog_manager.cc:1347] Generated new cluster ID: 282f861031a047619ab26d4cb30f7b16
I20240506 11:40:59.487071 19402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:40:59.493103 19402 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:40:59.494702 19402 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:40:59.504288 19402 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c9d18e1ec1d74d378dd4a7c374641ebe: Generated new TSK 0
I20240506 11:40:59.505264 19402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:40:59.515028 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45763
--builtin_ntp_servers=127.10.45.148:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20240506 11:40:59.823990 19415 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:40:59.824251 19415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:40:59.824314 19415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:40:59.824400 19415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:40:59.830838 19415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:40:59.830982 19415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:40:59.838490 19415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:45763
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:40:59.839582 19415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:40:59.841109 19415 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:40:59.851853 19420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:59.852707 19423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:40:59.853530 19421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:40:59.854367 19415 server_base.cc:1034] running on GCE node
I20240506 11:40:59.854866 19415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:40:59.855657 19415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:40:59.856848 19415 hybrid_clock.cc:648] HybridClock initialized: now 1714995659856814 us; error 27 us; skew 500 ppm
I20240506 11:40:59.857110 19415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:40:59.859043 19415 webserver.cc:460] Webserver started at http://127.10.45.129:32843/ using document root <none> and password file <none>
I20240506 11:40:59.859668 19415 fs_manager.cc:359] Metadata directory not provided
I20240506 11:40:59.859763 19415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:40:59.860038 19415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:40:59.862233 19415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
format_stamp: "Formatted at 2024-05-06 11:40:59 on dist-test-slave-7l0g"
I20240506 11:40:59.862924 19415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
format_stamp: "Formatted at 2024-05-06 11:40:59 on dist-test-slave-7l0g"
I20240506 11:40:59.867452 19415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240506 11:40:59.870306 19429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:59.871568 19415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:40:59.871708 19415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
format_stamp: "Formatted at 2024-05-06 11:40:59 on dist-test-slave-7l0g"
I20240506 11:40:59.871876 19415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:40:59.887527 19415 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:40:59.888249 19415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:40:59.888527 19415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:40:59.889335 19415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:40:59.890627 19415 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:40:59.890703 19415 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:59.890798 19415 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:40:59.890870 19415 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:40:59.920858 19415 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:44351
I20240506 11:40:59.920920 19541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:44351 every 8 connection(s)
I20240506 11:40:59.922067 19415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:40:59.928213 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 19415
I20240506 11:40:59.928380 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:40:59.933832 19542 heartbeater.cc:347] Connected to a master server at 127.10.45.190:45763
I20240506 11:40:59.934087 19542 heartbeater.cc:464] Registering TS with master...
I20240506 11:40:59.934820 19542 heartbeater.cc:511] Master 127.10.45.190:45763 requested a full tablet report, sending...
I20240506 11:40:59.936584 19356 ts_manager.cc:194] Registered new tserver with Master: 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129:44351)
I20240506 11:40:59.937912 19356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:42681
I20240506 11:40:59.943338 10422 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20240506 11:40:59.956506 10422 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20240506 11:40:59.956599 10422 test_util.cc:268] Using random seed: -1961205419
I20240506 11:40:59.972047 19356 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50558:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 11:40:59.998085 19477 tablet_service.cc:1467] Processing CreateTablet for tablet 98b01edf670741deac30eb0deaf9a531 (DEFAULT_TABLE table=test-workload [id=9a40e43e61b845e5bdf67e9ca8b04b8d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240506 11:40:59.998732 19470 tablet_service.cc:1467] Processing CreateTablet for tablet d930eb6b9e1a48a198bc48d3cb0d4d7e (DEFAULT_TABLE table=test-workload [id=9a40e43e61b845e5bdf67e9ca8b04b8d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240506 11:40:59.999179 19469 tablet_service.cc:1467] Processing CreateTablet for tablet f67127bcdaf24c1096f313e58eb4a7dc (DEFAULT_TABLE table=test-workload [id=9a40e43e61b845e5bdf67e9ca8b04b8d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240506 11:40:59.999256 19468 tablet_service.cc:1467] Processing CreateTablet for tablet c16fe3a355a44b7eb80a446104b5d92c (DEFAULT_TABLE table=test-workload [id=9a40e43e61b845e5bdf67e9ca8b04b8d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240506 11:40:59.998039 19472 tablet_service.cc:1467] Processing CreateTablet for tablet fb80f790884146bd97e7b88517061f21 (DEFAULT_TABLE table=test-workload [id=9a40e43e61b845e5bdf67e9ca8b04b8d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240506 11:40:59.999588 19473 tablet_service.cc:1467] Processing CreateTablet for tablet f893fb6e91d84af4824d8465a2ab186a (DEFAULT_TABLE table=test-workload [id=9a40e43e61b845e5bdf67e9ca8b04b8d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240506 11:41:00.000137 19471 tablet_service.cc:1467] Processing CreateTablet for tablet bbd3efaa685e49afaf05a6dbfd640208 (DEFAULT_TABLE table=test-workload [id=9a40e43e61b845e5bdf67e9ca8b04b8d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240506 11:41:00.000291 19477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b01edf670741deac30eb0deaf9a531. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:00.000615 19474 tablet_service.cc:1467] Processing CreateTablet for tablet dc32cf679b1c407689e92fb05f76961b (DEFAULT_TABLE table=test-workload [id=9a40e43e61b845e5bdf67e9ca8b04b8d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240506 11:41:00.000691 19473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f893fb6e91d84af4824d8465a2ab186a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:00.000974 19468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16fe3a355a44b7eb80a446104b5d92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:00.001116 19476 tablet_service.cc:1467] Processing CreateTablet for tablet 36cc1b49c11a454f9ab4be41ab94c5a7 (DEFAULT_TABLE table=test-workload [id=9a40e43e61b845e5bdf67e9ca8b04b8d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240506 11:41:00.001538 19476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cc1b49c11a454f9ab4be41ab94c5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:00.001753 19472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb80f790884146bd97e7b88517061f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:00.002135 19474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc32cf679b1c407689e92fb05f76961b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:00.002377 19469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67127bcdaf24c1096f313e58eb4a7dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:00.002578 19471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd3efaa685e49afaf05a6dbfd640208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:40:59.998126 19475 tablet_service.cc:1467] Processing CreateTablet for tablet 480d974932784a6a9efdd55b74002278 (DEFAULT_TABLE table=test-workload [id=9a40e43e61b845e5bdf67e9ca8b04b8d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240506 11:41:00.003281 19475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480d974932784a6a9efdd55b74002278. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:00.003619 19470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d930eb6b9e1a48a198bc48d3cb0d4d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:00.019402 19557 tablet_bootstrap.cc:492] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Bootstrap starting.
I20240506 11:41:00.023219 19557 tablet_bootstrap.cc:654] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:00.024428 19557 log.cc:824] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Log is configured to *not* fsync() on all Append() calls
I20240506 11:41:00.027330 19557 tablet_bootstrap.cc:492] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: No bootstrap required, opened a new log
I20240506 11:41:00.027542 19557 ts_tablet_manager.cc:1367] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240506 11:41:00.032707 19557 raft_consensus.cc:348] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.032979 19557 raft_consensus.cc:729] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Initialized, Role: FOLLOWER
I20240506 11:41:00.033358 19557 consensus_queue.cc:260] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.034219 19542 heartbeater.cc:503] Master 127.10.45.190:45763 was elected leader, sending a full tablet report...
I20240506 11:41:00.034574 19557 ts_tablet_manager.cc:1398] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240506 11:41:00.034912 19557 tablet_bootstrap.cc:492] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Bootstrap starting.
I20240506 11:41:00.036060 19557 tablet_bootstrap.cc:654] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:00.037588 19557 tablet_bootstrap.cc:492] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0: No bootstrap required, opened a new log
I20240506 11:41:00.037711 19557 ts_tablet_manager.cc:1367] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240506 11:41:00.038309 19557 raft_consensus.cc:348] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.038516 19557 raft_consensus.cc:729] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Initialized, Role: FOLLOWER
I20240506 11:41:00.038678 19557 consensus_queue.cc:260] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.039137 19557 ts_tablet_manager.cc:1398] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240506 11:41:00.039353 19557 tablet_bootstrap.cc:492] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Bootstrap starting.
I20240506 11:41:00.040782 19557 tablet_bootstrap.cc:654] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:00.042637 19557 tablet_bootstrap.cc:492] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0: No bootstrap required, opened a new log
I20240506 11:41:00.042757 19557 ts_tablet_manager.cc:1367] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240506 11:41:00.043330 19557 raft_consensus.cc:348] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.043460 19557 raft_consensus.cc:729] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Initialized, Role: FOLLOWER
I20240506 11:41:00.043617 19557 consensus_queue.cc:260] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.043956 19557 ts_tablet_manager.cc:1398] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240506 11:41:00.044201 19557 tablet_bootstrap.cc:492] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Bootstrap starting.
I20240506 11:41:00.045439 19557 tablet_bootstrap.cc:654] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:00.047147 19557 tablet_bootstrap.cc:492] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: No bootstrap required, opened a new log
I20240506 11:41:00.047282 19557 ts_tablet_manager.cc:1367] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240506 11:41:00.047770 19557 raft_consensus.cc:348] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.047914 19557 raft_consensus.cc:729] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Initialized, Role: FOLLOWER
I20240506 11:41:00.048072 19557 consensus_queue.cc:260] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.048470 19557 ts_tablet_manager.cc:1398] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:00.048678 19557 tablet_bootstrap.cc:492] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Bootstrap starting.
I20240506 11:41:00.049768 19557 tablet_bootstrap.cc:654] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:00.051363 19557 tablet_bootstrap.cc:492] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: No bootstrap required, opened a new log
I20240506 11:41:00.051505 19557 ts_tablet_manager.cc:1367] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240506 11:41:00.052109 19557 raft_consensus.cc:348] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.052302 19557 raft_consensus.cc:729] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Initialized, Role: FOLLOWER
I20240506 11:41:00.052455 19557 consensus_queue.cc:260] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.052820 19557 ts_tablet_manager.cc:1398] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:00.053078 19557 tablet_bootstrap.cc:492] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Bootstrap starting.
I20240506 11:41:00.054572 19557 tablet_bootstrap.cc:654] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:00.056278 19557 tablet_bootstrap.cc:492] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0: No bootstrap required, opened a new log
I20240506 11:41:00.056411 19557 ts_tablet_manager.cc:1367] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240506 11:41:00.056967 19557 raft_consensus.cc:348] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.057152 19557 raft_consensus.cc:729] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Initialized, Role: FOLLOWER
I20240506 11:41:00.057301 19557 consensus_queue.cc:260] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.057641 19557 ts_tablet_manager.cc:1398] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:00.057870 19557 tablet_bootstrap.cc:492] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Bootstrap starting.
I20240506 11:41:00.059222 19557 tablet_bootstrap.cc:654] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:00.060724 19557 tablet_bootstrap.cc:492] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0: No bootstrap required, opened a new log
I20240506 11:41:00.060863 19557 ts_tablet_manager.cc:1367] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240506 11:41:00.061394 19557 raft_consensus.cc:348] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.061528 19557 raft_consensus.cc:729] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Initialized, Role: FOLLOWER
I20240506 11:41:00.061692 19557 consensus_queue.cc:260] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.062024 19557 ts_tablet_manager.cc:1398] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:00.062209 19557 tablet_bootstrap.cc:492] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Bootstrap starting.
I20240506 11:41:00.063416 19557 tablet_bootstrap.cc:654] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:00.064927 19557 tablet_bootstrap.cc:492] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: No bootstrap required, opened a new log
I20240506 11:41:00.065054 19557 ts_tablet_manager.cc:1367] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240506 11:41:00.065657 19557 raft_consensus.cc:348] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.065830 19557 raft_consensus.cc:729] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Initialized, Role: FOLLOWER
I20240506 11:41:00.065974 19557 consensus_queue.cc:260] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.066370 19557 ts_tablet_manager.cc:1398] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:00.066695 19557 tablet_bootstrap.cc:492] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Bootstrap starting.
I20240506 11:41:00.067689 19557 tablet_bootstrap.cc:654] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:00.069460 19557 tablet_bootstrap.cc:492] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0: No bootstrap required, opened a new log
I20240506 11:41:00.069607 19557 ts_tablet_manager.cc:1367] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240506 11:41:00.070258 19557 raft_consensus.cc:348] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.070447 19557 raft_consensus.cc:729] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Initialized, Role: FOLLOWER
I20240506 11:41:00.070605 19557 consensus_queue.cc:260] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.071009 19557 ts_tablet_manager.cc:1398] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240506 11:41:00.071282 19557 tablet_bootstrap.cc:492] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Bootstrap starting.
I20240506 11:41:00.072432 19557 tablet_bootstrap.cc:654] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:00.073933 19557 tablet_bootstrap.cc:492] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: No bootstrap required, opened a new log
I20240506 11:41:00.074081 19557 ts_tablet_manager.cc:1367] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240506 11:41:00.074679 19557 raft_consensus.cc:348] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.074796 19557 raft_consensus.cc:729] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Initialized, Role: FOLLOWER
I20240506 11:41:00.074908 19557 consensus_queue.cc:260] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.075273 19557 ts_tablet_manager.cc:1398] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240506 11:41:00.115760 19497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "480d974932784a6a9efdd55b74002278"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.116040 19497 raft_consensus.cc:482] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:00.116227 19497 raft_consensus.cc:3037] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:00.118140 19497 raft_consensus.cc:504] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.118644 19497 leader_election.cc:302] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df7b6eb4024a0ea0d84d7b6b16e0a0; no voters:
I20240506 11:41:00.118999 19497 leader_election.cc:288] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:00.119050 19559 raft_consensus.cc:2781] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:00.119597 19559 raft_consensus.cc:686] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.120136 19559 consensus_queue.cc:237] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.123577 19356 catalog_manager.cc:5484] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } health_report { overall_health: HEALTHY } } }
I20240506 11:41:00.131254 19497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98b01edf670741deac30eb0deaf9a531"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.131433 19497 raft_consensus.cc:482] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:00.131531 19497 raft_consensus.cc:3037] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:00.133152 19497 raft_consensus.cc:504] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.133340 19497 leader_election.cc:302] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df7b6eb4024a0ea0d84d7b6b16e0a0; no voters:
I20240506 11:41:00.133494 19497 leader_election.cc:288] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:00.133551 19559 raft_consensus.cc:2781] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:00.133723 19559 raft_consensus.cc:686] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.133971 19559 consensus_queue.cc:237] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.136067 19356 catalog_manager.cc:5484] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } health_report { overall_health: HEALTHY } } }
I20240506 11:41:00.142482 19497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c16fe3a355a44b7eb80a446104b5d92c"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.142622 19497 raft_consensus.cc:482] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:00.142720 19497 raft_consensus.cc:3037] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:00.144438 19497 raft_consensus.cc:504] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.144599 19497 leader_election.cc:302] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df7b6eb4024a0ea0d84d7b6b16e0a0; no voters:
I20240506 11:41:00.144719 19497 leader_election.cc:288] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:00.144816 19559 raft_consensus.cc:2781] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:00.144942 19559 raft_consensus.cc:686] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.145195 19559 consensus_queue.cc:237] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.147233 19356 catalog_manager.cc:5484] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } health_report { overall_health: HEALTHY } } }
I20240506 11:41:00.154003 19497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f67127bcdaf24c1096f313e58eb4a7dc"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.154151 19497 raft_consensus.cc:482] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:00.154249 19497 raft_consensus.cc:3037] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:00.155831 19497 raft_consensus.cc:504] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.156023 19497 leader_election.cc:302] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df7b6eb4024a0ea0d84d7b6b16e0a0; no voters:
I20240506 11:41:00.156164 19497 leader_election.cc:288] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:00.156229 19559 raft_consensus.cc:2781] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:00.156389 19559 raft_consensus.cc:686] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.156633 19559 consensus_queue.cc:237] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.158589 19356 catalog_manager.cc:5484] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } health_report { overall_health: HEALTHY } } }
I20240506 11:41:00.165192 19497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb80f790884146bd97e7b88517061f21"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.165329 19497 raft_consensus.cc:482] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:00.165437 19497 raft_consensus.cc:3037] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:00.167279 19497 raft_consensus.cc:504] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.167497 19497 leader_election.cc:302] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df7b6eb4024a0ea0d84d7b6b16e0a0; no voters:
I20240506 11:41:00.167680 19497 leader_election.cc:288] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:00.167755 19559 raft_consensus.cc:2781] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:00.167927 19559 raft_consensus.cc:686] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.168164 19559 consensus_queue.cc:237] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.170411 19356 catalog_manager.cc:5484] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } health_report { overall_health: HEALTHY } } }
W20240506 11:41:00.173693 19543 tablet.cc:2369] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:41:00.178165 19497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bbd3efaa685e49afaf05a6dbfd640208"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.178285 19497 raft_consensus.cc:482] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:00.178359 19497 raft_consensus.cc:3037] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:00.180122 19497 raft_consensus.cc:504] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.180316 19497 leader_election.cc:302] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df7b6eb4024a0ea0d84d7b6b16e0a0; no voters:
I20240506 11:41:00.180446 19497 leader_election.cc:288] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:00.180526 19559 raft_consensus.cc:2781] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:00.180685 19559 raft_consensus.cc:686] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.181103 19559 consensus_queue.cc:237] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.183197 19356 catalog_manager.cc:5484] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } health_report { overall_health: HEALTHY } } }
I20240506 11:41:00.189675 19497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f893fb6e91d84af4824d8465a2ab186a"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.189828 19497 raft_consensus.cc:482] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:00.189957 19497 raft_consensus.cc:3037] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:00.191638 19497 raft_consensus.cc:504] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.191844 19497 leader_election.cc:302] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df7b6eb4024a0ea0d84d7b6b16e0a0; no voters:
I20240506 11:41:00.191999 19497 leader_election.cc:288] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:00.192111 19559 raft_consensus.cc:2781] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:00.192282 19559 raft_consensus.cc:686] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.192549 19559 consensus_queue.cc:237] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.194638 19356 catalog_manager.cc:5484] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } health_report { overall_health: HEALTHY } } }
I20240506 11:41:00.201284 19497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc32cf679b1c407689e92fb05f76961b"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.201431 19497 raft_consensus.cc:482] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:00.201529 19497 raft_consensus.cc:3037] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:00.203125 19497 raft_consensus.cc:504] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.203332 19497 leader_election.cc:302] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df7b6eb4024a0ea0d84d7b6b16e0a0; no voters:
I20240506 11:41:00.203509 19497 leader_election.cc:288] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:00.203576 19559 raft_consensus.cc:2781] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:00.203745 19559 raft_consensus.cc:686] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.203981 19559 consensus_queue.cc:237] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.206079 19356 catalog_manager.cc:5484] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } health_report { overall_health: HEALTHY } } }
I20240506 11:41:00.212756 19497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "36cc1b49c11a454f9ab4be41ab94c5a7"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.212894 19497 raft_consensus.cc:482] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:00.213007 19497 raft_consensus.cc:3037] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:00.214687 19497 raft_consensus.cc:504] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.214879 19497 leader_election.cc:302] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df7b6eb4024a0ea0d84d7b6b16e0a0; no voters:
I20240506 11:41:00.215067 19497 leader_election.cc:288] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:00.215158 19559 raft_consensus.cc:2781] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:00.215358 19559 raft_consensus.cc:686] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.215581 19559 consensus_queue.cc:237] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.217471 19356 catalog_manager.cc:5484] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } health_report { overall_health: HEALTHY } } }
I20240506 11:41:00.224328 19497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d930eb6b9e1a48a198bc48d3cb0d4d7e"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.224479 19497 raft_consensus.cc:482] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:00.224570 19497 raft_consensus.cc:3037] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:00.226044 19497 raft_consensus.cc:504] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.226220 19497 leader_election.cc:302] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26df7b6eb4024a0ea0d84d7b6b16e0a0; no voters:
I20240506 11:41:00.226337 19497 leader_election.cc:288] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:00.226402 19559 raft_consensus.cc:2781] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:00.226608 19559 raft_consensus.cc:686] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.226823 19559 consensus_queue.cc:237] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.228760 19356 catalog_manager.cc:5484] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26df7b6eb4024a0ea0d84d7b6b16e0a0 (127.10.45.129). New cstate: current_term: 1 leader_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } health_report { overall_health: HEALTHY } } }
I20240506 11:41:00.235662 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet 480d974932784a6a9efdd55b74002278 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.236341 19583 tablet_replica.cc:330] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: stopping tablet replica
I20240506 11:41:00.236681 19583 raft_consensus.cc:2229] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Raft consensus shutting down.
I20240506 11:41:00.236922 19583 raft_consensus.cc:2258] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:00.238128 19583 ts_tablet_manager.cc:1875] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:00.241992 19583 ts_tablet_manager.cc:1888] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:00.242107 19583 log.cc:1196] T 480d974932784a6a9efdd55b74002278 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/480d974932784a6a9efdd55b74002278
I20240506 11:41:00.249886 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b01edf670741deac30eb0deaf9a531 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.250114 19583 tablet_replica.cc:330] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: stopping tablet replica
I20240506 11:41:00.250268 19583 raft_consensus.cc:2229] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Raft consensus shutting down.
I20240506 11:41:00.250437 19583 raft_consensus.cc:2258] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:00.251154 19583 ts_tablet_manager.cc:1875] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:00.254601 19583 ts_tablet_manager.cc:1888] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:00.254691 19583 log.cc:1196] T 98b01edf670741deac30eb0deaf9a531 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/98b01edf670741deac30eb0deaf9a531
I20240506 11:41:00.262449 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet c16fe3a355a44b7eb80a446104b5d92c with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.262703 19583 tablet_replica.cc:330] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0: stopping tablet replica
I20240506 11:41:00.262876 19583 raft_consensus.cc:2229] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Raft consensus shutting down.
I20240506 11:41:00.263075 19583 raft_consensus.cc:2258] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:00.263782 19583 ts_tablet_manager.cc:1875] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:00.267516 19583 ts_tablet_manager.cc:1888] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:00.267604 19583 log.cc:1196] T c16fe3a355a44b7eb80a446104b5d92c P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/c16fe3a355a44b7eb80a446104b5d92c
I20240506 11:41:00.275193 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet f67127bcdaf24c1096f313e58eb4a7dc with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.275417 19583 tablet_replica.cc:330] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0: stopping tablet replica
I20240506 11:41:00.275575 19583 raft_consensus.cc:2229] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Raft consensus shutting down.
I20240506 11:41:00.275736 19583 raft_consensus.cc:2258] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:00.276445 19583 ts_tablet_manager.cc:1875] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:00.279986 19583 ts_tablet_manager.cc:1888] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:00.280102 19583 log.cc:1196] T f67127bcdaf24c1096f313e58eb4a7dc P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/f67127bcdaf24c1096f313e58eb4a7dc
I20240506 11:41:00.287493 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet fb80f790884146bd97e7b88517061f21 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.287724 19583 tablet_replica.cc:330] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: stopping tablet replica
I20240506 11:41:00.287891 19583 raft_consensus.cc:2229] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Raft consensus shutting down.
I20240506 11:41:00.288056 19583 raft_consensus.cc:2258] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:00.288810 19583 ts_tablet_manager.cc:1875] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:00.292234 19583 ts_tablet_manager.cc:1888] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:00.292340 19583 log.cc:1196] T fb80f790884146bd97e7b88517061f21 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/fb80f790884146bd97e7b88517061f21
I20240506 11:41:00.300172 19497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bbd3efaa685e49afaf05a6dbfd640208"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.300396 19497 raft_consensus.cc:580] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Received request to step down
I20240506 11:41:00.300496 19497 raft_consensus.cc:3032] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:41:00.300572 19497 raft_consensus.cc:729] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.300956 19497 consensus_queue.cc:260] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.301127 19497 raft_consensus.cc:3037] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:41:00.308698 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd3efaa685e49afaf05a6dbfd640208 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.308933 19583 tablet_replica.cc:330] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: stopping tablet replica
I20240506 11:41:00.309104 19583 raft_consensus.cc:2229] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:41:00.309278 19583 raft_consensus.cc:2258] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:00.309993 19583 ts_tablet_manager.cc:1875] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:00.313578 19583 ts_tablet_manager.cc:1888] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:00.313692 19583 log.cc:1196] T bbd3efaa685e49afaf05a6dbfd640208 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/bbd3efaa685e49afaf05a6dbfd640208
I20240506 11:41:00.321367 19497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f893fb6e91d84af4824d8465a2ab186a"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.321511 19497 raft_consensus.cc:580] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Received request to step down
I20240506 11:41:00.321596 19497 raft_consensus.cc:3032] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:41:00.321657 19497 raft_consensus.cc:729] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.321877 19497 consensus_queue.cc:260] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.322000 19497 raft_consensus.cc:3037] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:41:00.329005 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet f893fb6e91d84af4824d8465a2ab186a with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.329233 19583 tablet_replica.cc:330] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0: stopping tablet replica
I20240506 11:41:00.329388 19583 raft_consensus.cc:2229] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:41:00.329546 19583 raft_consensus.cc:2258] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:00.330243 19583 ts_tablet_manager.cc:1875] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:00.333950 19583 ts_tablet_manager.cc:1888] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:00.334077 19583 log.cc:1196] T f893fb6e91d84af4824d8465a2ab186a P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/f893fb6e91d84af4824d8465a2ab186a
I20240506 11:41:00.341696 19497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc32cf679b1c407689e92fb05f76961b"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.341852 19497 raft_consensus.cc:580] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Received request to step down
I20240506 11:41:00.341948 19497 raft_consensus.cc:3032] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:41:00.342011 19497 raft_consensus.cc:729] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.342208 19497 consensus_queue.cc:260] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.342322 19497 raft_consensus.cc:3037] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:41:00.349804 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet dc32cf679b1c407689e92fb05f76961b with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.350028 19583 tablet_replica.cc:330] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0: stopping tablet replica
I20240506 11:41:00.350179 19583 raft_consensus.cc:2229] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:41:00.350343 19583 raft_consensus.cc:2258] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:00.351243 19583 ts_tablet_manager.cc:1875] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:00.355265 19583 ts_tablet_manager.cc:1888] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:00.355376 19583 log.cc:1196] T dc32cf679b1c407689e92fb05f76961b P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc32cf679b1c407689e92fb05f76961b
I20240506 11:41:00.363507 19497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36cc1b49c11a454f9ab4be41ab94c5a7"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.363667 19497 raft_consensus.cc:580] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Received request to step down
I20240506 11:41:00.363746 19497 raft_consensus.cc:3032] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:41:00.363791 19497 raft_consensus.cc:729] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.363947 19497 consensus_queue.cc:260] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.364058 19497 raft_consensus.cc:3037] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:41:00.371531 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet 36cc1b49c11a454f9ab4be41ab94c5a7 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.371771 19583 tablet_replica.cc:330] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: stopping tablet replica
I20240506 11:41:00.371924 19583 raft_consensus.cc:2229] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:41:00.372081 19583 raft_consensus.cc:2258] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:00.372821 19583 ts_tablet_manager.cc:1875] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:00.376611 19583 ts_tablet_manager.cc:1888] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:00.376698 19583 log.cc:1196] T 36cc1b49c11a454f9ab4be41ab94c5a7 P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/36cc1b49c11a454f9ab4be41ab94c5a7
I20240506 11:41:00.384754 19497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d930eb6b9e1a48a198bc48d3cb0d4d7e"
dest_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0"
from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.384910 19497 raft_consensus.cc:580] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Received request to step down
I20240506 11:41:00.384984 19497 raft_consensus.cc:3032] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:41:00.385026 19497 raft_consensus.cc:729] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26df7b6eb4024a0ea0d84d7b6b16e0a0, State: Running, Role: LEADER
I20240506 11:41:00.385190 19497 consensus_queue.cc:260] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26df7b6eb4024a0ea0d84d7b6b16e0a0" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 44351 } }
I20240506 11:41:00.385289 19497 raft_consensus.cc:3037] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:41:00.392199 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet d930eb6b9e1a48a198bc48d3cb0d4d7e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:38138
I20240506 11:41:00.392449 19583 tablet_replica.cc:330] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0: stopping tablet replica
I20240506 11:41:00.392572 19583 raft_consensus.cc:2229] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:41:00.392724 19583 raft_consensus.cc:2258] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:00.393512 19583 ts_tablet_manager.cc:1875] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:00.397163 19583 ts_tablet_manager.cc:1888] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:00.397253 19583 log.cc:1196] T d930eb6b9e1a48a198bc48d3cb0d4d7e P 26df7b6eb4024a0ea0d84d7b6b16e0a0: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/d930eb6b9e1a48a198bc48d3cb0d4d7e
I20240506 11:41:00.405947 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 19415
I20240506 11:41:00.411773 19537 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:41:00.512455 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 19415
I20240506 11:41:00.530521 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 19325
I20240506 11:41:00.531570 19386 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:41:00.626505 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 19325
2024-05-06T11:41:00Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0 (1656 ms)
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1
I20240506 11:41:00.662565 10422 test_util.cc:268] Using random seed: -1960499438
I20240506 11:41:00.664369 10422 ts_itest-base.cc:112] Starting cluster with:
I20240506 11:41:00.664430 10422 ts_itest-base.cc:113] --------------
I20240506 11:41:00.664482 10422 ts_itest-base.cc:114] 3 tablet servers
I20240506 11:41:00.664526 10422 ts_itest-base.cc:115] 3 replicas per TS
I20240506 11:41:00.664588 10422 ts_itest-base.cc:116] --------------
2024-05-06T11:41:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T11:41:00Z Disabled control of system clock
I20240506 11:41:00.697549 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:37625
--webserver_interface=127.10.45.190
--webserver_port=0
--builtin_ntp_servers=127.10.45.148:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.45.190:37625
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20240506 11:41:00.991518 19594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:41:00.991782 19594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:41:00.991865 19594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:41:00.998355 19594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 11:41:00.998445 19594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:41:00.998486 19594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 11:41:00.998538 19594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 11:41:01.006100 19594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.45.190:37625
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.45.190:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.45.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:41:01.007349 19594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:41:01.008899 19594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:41:01.019058 19599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:41:01.019798 19602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:41:01.020639 19600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:41:01.021451 19594 server_base.cc:1034] running on GCE node
I20240506 11:41:01.022110 19594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:41:01.023387 19594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:41:01.024590 19594 hybrid_clock.cc:648] HybridClock initialized: now 1714995661024535 us; error 29 us; skew 500 ppm
I20240506 11:41:01.024879 19594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:41:01.026857 19594 webserver.cc:460] Webserver started at http://127.10.45.190:36391/ using document root <none> and password file <none>
I20240506 11:41:01.027473 19594 fs_manager.cc:359] Metadata directory not provided
I20240506 11:41:01.027590 19594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:41:01.027861 19594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:41:01.030108 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "71efb7fa8d5e4fe0b393e754e267efbd"
format_stamp: "Formatted at 2024-05-06 11:41:01 on dist-test-slave-7l0g"
I20240506 11:41:01.030766 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "71efb7fa8d5e4fe0b393e754e267efbd"
format_stamp: "Formatted at 2024-05-06 11:41:01 on dist-test-slave-7l0g"
I20240506 11:41:01.034986 19594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240506 11:41:01.037655 19608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:41:01.038791 19594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240506 11:41:01.038983 19594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
uuid: "71efb7fa8d5e4fe0b393e754e267efbd"
format_stamp: "Formatted at 2024-05-06 11:41:01 on dist-test-slave-7l0g"
I20240506 11:41:01.039144 19594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:41:01.073227 19594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:41:01.073876 19594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:41:01.074133 19594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:41:01.089895 19594 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.190:37625
I20240506 11:41:01.089926 19659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.190:37625 every 8 connection(s)
I20240506 11:41:01.091014 19594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240506 11:41:01.094345 19660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:01.095523 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 19594
I20240506 11:41:01.095671 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240506 11:41:01.103245 19660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd: Bootstrap starting.
I20240506 11:41:01.105979 19660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:01.107031 19660 log.cc:824] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd: Log is configured to *not* fsync() on all Append() calls
I20240506 11:41:01.109175 19660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd: No bootstrap required, opened a new log
I20240506 11:41:01.114473 19660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71efb7fa8d5e4fe0b393e754e267efbd" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37625 } }
I20240506 11:41:01.114740 19660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 11:41:01.114821 19660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71efb7fa8d5e4fe0b393e754e267efbd, State: Initialized, Role: FOLLOWER
I20240506 11:41:01.115332 19660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71efb7fa8d5e4fe0b393e754e267efbd" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37625 } }
I20240506 11:41:01.115507 19660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 11:41:01.115614 19660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 11:41:01.115739 19660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:01.117609 19660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71efb7fa8d5e4fe0b393e754e267efbd" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37625 } }
I20240506 11:41:01.118017 19660 leader_election.cc:302] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71efb7fa8d5e4fe0b393e754e267efbd; no voters:
I20240506 11:41:01.118606 19660 leader_election.cc:288] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 11:41:01.118736 19665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:01.119437 19665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [term 1 LEADER]: Becoming Leader. State: Replica: 71efb7fa8d5e4fe0b393e754e267efbd, State: Running, Role: LEADER
I20240506 11:41:01.119936 19665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71efb7fa8d5e4fe0b393e754e267efbd" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37625 } }
I20240506 11:41:01.120434 19660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [sys.catalog]: configured and running, proceeding with master startup.
I20240506 11:41:01.122408 19667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71efb7fa8d5e4fe0b393e754e267efbd. Latest consensus state: current_term: 1 leader_uuid: "71efb7fa8d5e4fe0b393e754e267efbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71efb7fa8d5e4fe0b393e754e267efbd" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37625 } } }
I20240506 11:41:01.122725 19667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [sys.catalog]: This master's current role is: LEADER
I20240506 11:41:01.122388 19666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71efb7fa8d5e4fe0b393e754e267efbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71efb7fa8d5e4fe0b393e754e267efbd" member_type: VOTER last_known_addr { host: "127.10.45.190" port: 37625 } } }
I20240506 11:41:01.123071 19666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd [sys.catalog]: This master's current role is: LEADER
I20240506 11:41:01.124511 19670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 11:41:01.128793 19670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 11:41:01.135380 19670 catalog_manager.cc:1347] Generated new cluster ID: 98303ebac15e43bb8a54f84920fe99c2
I20240506 11:41:01.135489 19670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 11:41:01.147284 19670 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 11:41:01.148232 19670 catalog_manager.cc:1504] Loading token signing keys...
I20240506 11:41:01.154999 19670 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 71efb7fa8d5e4fe0b393e754e267efbd: Generated new TSK 0
I20240506 11:41:01.155616 19670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 11:41:01.167877 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.129:0
--local_ip_for_outbound_sockets=127.10.45.129
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:37625
--builtin_ntp_servers=127.10.45.148:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20240506 11:41:01.479346 19684 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:41:01.479661 19684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:41:01.479753 19684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:41:01.479876 19684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:41:01.486996 19684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:41:01.487192 19684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.129
I20240506 11:41:01.495558 19684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.45.129
--webserver_port=0
--tserver_master_addrs=127.10.45.190:37625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.129
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:41:01.496783 19684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:41:01.498416 19684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:41:01.510706 19689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:41:01.511607 19690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:41:01.512485 19692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:41:01.513312 19684 server_base.cc:1034] running on GCE node
I20240506 11:41:01.513902 19684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:41:01.514786 19684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:41:01.515981 19684 hybrid_clock.cc:648] HybridClock initialized: now 1714995661515931 us; error 44 us; skew 500 ppm
I20240506 11:41:01.516269 19684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:41:01.518401 19684 webserver.cc:460] Webserver started at http://127.10.45.129:45819/ using document root <none> and password file <none>
I20240506 11:41:01.519146 19684 fs_manager.cc:359] Metadata directory not provided
I20240506 11:41:01.519258 19684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:41:01.519567 19684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:41:01.522014 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "83434b0d0851482f95a9eaebcefb794d"
format_stamp: "Formatted at 2024-05-06 11:41:01 on dist-test-slave-7l0g"
I20240506 11:41:01.522748 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "83434b0d0851482f95a9eaebcefb794d"
format_stamp: "Formatted at 2024-05-06 11:41:01 on dist-test-slave-7l0g"
I20240506 11:41:01.527451 19684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240506 11:41:01.530406 19698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:41:01.531620 19684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240506 11:41:01.531770 19684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "83434b0d0851482f95a9eaebcefb794d"
format_stamp: "Formatted at 2024-05-06 11:41:01 on dist-test-slave-7l0g"
I20240506 11:41:01.531965 19684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:41:01.565932 19684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:41:01.566802 19684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:41:01.567085 19684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:41:01.568076 19684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:41:01.569576 19684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:41:01.569660 19684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:41:01.569766 19684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:41:01.569849 19684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:41:01.597838 19684 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.129:43585
I20240506 11:41:01.597913 19810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.129:43585 every 8 connection(s)
I20240506 11:41:01.599114 19684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240506 11:41:01.603629 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 19684
I20240506 11:41:01.603823 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240506 11:41:01.607175 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.130:0
--local_ip_for_outbound_sockets=127.10.45.130
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:37625
--builtin_ntp_servers=127.10.45.148:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20240506 11:41:01.612727 19811 heartbeater.cc:347] Connected to a master server at 127.10.45.190:37625
I20240506 11:41:01.613052 19811 heartbeater.cc:464] Registering TS with master...
I20240506 11:41:01.614034 19811 heartbeater.cc:511] Master 127.10.45.190:37625 requested a full tablet report, sending...
I20240506 11:41:01.616228 19625 ts_manager.cc:194] Registered new tserver with Master: 83434b0d0851482f95a9eaebcefb794d (127.10.45.129:43585)
I20240506 11:41:01.617637 19625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.129:56567
W20240506 11:41:01.926891 19814 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:41:01.927208 19814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:41:01.927294 19814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:41:01.927417 19814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:41:01.934398 19814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:41:01.934569 19814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.130
I20240506 11:41:01.943128 19814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.45.130
--webserver_port=0
--tserver_master_addrs=127.10.45.190:37625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.130
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:41:01.944382 19814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:41:01.946046 19814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:41:01.957469 19819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:41:01.958261 19822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:41:01.959062 19820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:41:01.959935 19814 server_base.cc:1034] running on GCE node
I20240506 11:41:01.960497 19814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:41:01.961421 19814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:41:01.962618 19814 hybrid_clock.cc:648] HybridClock initialized: now 1714995661962575 us; error 39 us; skew 500 ppm
I20240506 11:41:01.962932 19814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:41:01.965062 19814 webserver.cc:460] Webserver started at http://127.10.45.130:39827/ using document root <none> and password file <none>
I20240506 11:41:01.965759 19814 fs_manager.cc:359] Metadata directory not provided
I20240506 11:41:01.965880 19814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:41:01.966159 19814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:41:01.968561 19814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
format_stamp: "Formatted at 2024-05-06 11:41:01 on dist-test-slave-7l0g"
I20240506 11:41:01.969326 19814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
format_stamp: "Formatted at 2024-05-06 11:41:01 on dist-test-slave-7l0g"
I20240506 11:41:01.973876 19814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240506 11:41:01.976924 19828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:41:01.978219 19814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240506 11:41:01.978370 19814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
format_stamp: "Formatted at 2024-05-06 11:41:01 on dist-test-slave-7l0g"
I20240506 11:41:01.978551 19814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:41:01.998163 19814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:41:01.999004 19814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:41:01.999264 19814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:41:02.000173 19814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:41:02.001560 19814 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:41:02.001660 19814 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:41:02.001778 19814 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:41:02.001859 19814 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:41:02.032855 19814 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.130:41169
I20240506 11:41:02.032886 19940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.130:41169 every 8 connection(s)
I20240506 11:41:02.034137 19814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240506 11:41:02.043857 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 19814
I20240506 11:41:02.044018 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240506 11:41:02.046728 19941 heartbeater.cc:347] Connected to a master server at 127.10.45.190:37625
I20240506 11:41:02.047060 10422 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
/tmp/dist-test-taskNkhkL1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.45.131:0
--local_ip_for_outbound_sockets=127.10.45.131
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:37625
--builtin_ntp_servers=127.10.45.148:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20240506 11:41:02.047116 19941 heartbeater.cc:464] Registering TS with master...
I20240506 11:41:02.048084 19941 heartbeater.cc:511] Master 127.10.45.190:37625 requested a full tablet report, sending...
I20240506 11:41:02.049503 19625 ts_manager.cc:194] Registered new tserver with Master: 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169)
I20240506 11:41:02.050202 19625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.130:53427
W20240506 11:41:02.362788 19944 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240506 11:41:02.363137 19944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 11:41:02.363231 19944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 11:41:02.363346 19944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 11:41:02.370640 19944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 11:41:02.370817 19944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.45.131
I20240506 11:41:02.379350 19944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.45.148:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.45.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.45.131
--webserver_port=0
--tserver_master_addrs=127.10.45.190:37625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.45.131
--log_dir=/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 11:18:00 UTC on efd1337a2aa3
build id 117
ASAN enabled
I20240506 11:41:02.380518 19944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 11:41:02.382068 19944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 11:41:02.393651 19952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:41:02.394508 19950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 11:41:02.395277 19949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 11:41:02.396139 19944 server_base.cc:1034] running on GCE node
I20240506 11:41:02.396664 19944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 11:41:02.397521 19944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 11:41:02.398710 19944 hybrid_clock.cc:648] HybridClock initialized: now 1714995662398663 us; error 39 us; skew 500 ppm
I20240506 11:41:02.399034 19944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 11:41:02.401101 19944 webserver.cc:460] Webserver started at http://127.10.45.131:44191/ using document root <none> and password file <none>
I20240506 11:41:02.401908 19944 fs_manager.cc:359] Metadata directory not provided
I20240506 11:41:02.402031 19944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 11:41:02.402346 19944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 11:41:02.404762 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5414c092bba046b69d1cb7a07bc7e48e"
format_stamp: "Formatted at 2024-05-06 11:41:02 on dist-test-slave-7l0g"
I20240506 11:41:02.405501 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5414c092bba046b69d1cb7a07bc7e48e"
format_stamp: "Formatted at 2024-05-06 11:41:02 on dist-test-slave-7l0g"
I20240506 11:41:02.410513 19944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240506 11:41:02.413741 19958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 11:41:02.414992 19944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240506 11:41:02.415135 19944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5414c092bba046b69d1cb7a07bc7e48e"
format_stamp: "Formatted at 2024-05-06 11:41:02 on dist-test-slave-7l0g"
I20240506 11:41:02.415311 19944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 11:41:02.437637 19944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 11:41:02.438419 19944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 11:41:02.438670 19944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 11:41:02.439615 19944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 11:41:02.441092 19944 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 11:41:02.441187 19944 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 11:41:02.441296 19944 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 11:41:02.441370 19944 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 11:41:02.471788 19944 rpc_server.cc:307] RPC server started. Bound to: 127.10.45.131:44397
I20240506 11:41:02.471860 20070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.45.131:44397 every 8 connection(s)
I20240506 11:41:02.472976 19944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240506 11:41:02.481820 10422 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu as pid 19944
I20240506 11:41:02.482015 10422 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240506 11:41:02.485057 20071 heartbeater.cc:347] Connected to a master server at 127.10.45.190:37625
I20240506 11:41:02.485352 20071 heartbeater.cc:464] Registering TS with master...
I20240506 11:41:02.486135 20071 heartbeater.cc:511] Master 127.10.45.190:37625 requested a full tablet report, sending...
I20240506 11:41:02.487375 19625 ts_manager.cc:194] Registered new tserver with Master: 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397)
I20240506 11:41:02.488070 19625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.45.131:54457
I20240506 11:41:02.497225 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:41:02.511294 10422 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240506 11:41:02.511379 10422 test_util.cc:268] Using random seed: -1958650618
I20240506 11:41:02.527253 19625 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43924:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240506 11:41:02.529147 19625 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240506 11:41:02.566180 19873 tablet_service.cc:1467] Processing CreateTablet for tablet dee074a7650241048c409bd298d57f43 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240506 11:41:02.566228 19876 tablet_service.cc:1467] Processing CreateTablet for tablet 481042567046471aac1926a16d3a1265 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240506 11:41:02.567278 19872 tablet_service.cc:1467] Processing CreateTablet for tablet f7cd7fd9a7a947fb87505997aef18eb4 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240506 11:41:02.567922 19870 tablet_service.cc:1467] Processing CreateTablet for tablet 605bbc120f734820951ecb54606dfde6 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240506 11:41:02.568270 19873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee074a7650241048c409bd298d57f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.568715 19868 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1c54d70c2448d1b4a18a6bdcc91eb3 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240506 11:41:02.569133 19868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1c54d70c2448d1b4a18a6bdcc91eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.569027 19742 tablet_service.cc:1467] Processing CreateTablet for tablet f7cd7fd9a7a947fb87505997aef18eb4 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240506 11:41:02.570427 19743 tablet_service.cc:1467] Processing CreateTablet for tablet dee074a7650241048c409bd298d57f43 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240506 11:41:02.570762 19742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7cd7fd9a7a947fb87505997aef18eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.569424 19744 tablet_service.cc:1467] Processing CreateTablet for tablet 76cd0c3599094204a8bbfe14e296a750 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240506 11:41:02.570955 19869 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc17ce9c9a84dffa7cfa63adbdfce3e (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240506 11:41:02.571266 19746 tablet_service.cc:1467] Processing CreateTablet for tablet 481042567046471aac1926a16d3a1265 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240506 11:41:02.571528 19867 tablet_service.cc:1467] Processing CreateTablet for tablet 90737f5ce5dd4a4d8511d35ac96a0bcd (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240506 11:41:02.571529 19876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481042567046471aac1926a16d3a1265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.572105 19875 tablet_service.cc:1467] Processing CreateTablet for tablet 83b67259f0024c23a55fb1ad46d21fe6 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240506 11:41:02.572554 19875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b67259f0024c23a55fb1ad46d21fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.566603 19874 tablet_service.cc:1467] Processing CreateTablet for tablet 76cd0c3599094204a8bbfe14e296a750 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240506 11:41:02.575156 19744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76cd0c3599094204a8bbfe14e296a750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.575454 19874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76cd0c3599094204a8bbfe14e296a750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.578466 19746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481042567046471aac1926a16d3a1265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.579007 19745 tablet_service.cc:1467] Processing CreateTablet for tablet 83b67259f0024c23a55fb1ad46d21fe6 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240506 11:41:02.579526 19745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b67259f0024c23a55fb1ad46d21fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.583158 19743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee074a7650241048c409bd298d57f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.587229 19870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 605bbc120f734820951ecb54606dfde6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.591269 19872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7cd7fd9a7a947fb87505997aef18eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.594337 20095 tablet_bootstrap.cc:492] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: Bootstrap starting.
I20240506 11:41:02.595155 19869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc17ce9c9a84dffa7cfa63adbdfce3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.597980 19741 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1c54d70c2448d1b4a18a6bdcc91eb3 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240506 11:41:02.598132 19740 tablet_service.cc:1467] Processing CreateTablet for tablet 90737f5ce5dd4a4d8511d35ac96a0bcd (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240506 11:41:02.598469 19741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1c54d70c2448d1b4a18a6bdcc91eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.598371 19871 tablet_service.cc:1467] Processing CreateTablet for tablet c9423c59183d4a12a563adfe94f68648 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240506 11:41:02.598743 19740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90737f5ce5dd4a4d8511d35ac96a0bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.598896 19871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9423c59183d4a12a563adfe94f68648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.599123 19744 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc17ce9c9a84dffa7cfa63adbdfce3e (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240506 11:41:02.599453 19744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc17ce9c9a84dffa7cfa63adbdfce3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.602433 19867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90737f5ce5dd4a4d8511d35ac96a0bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.609161 19746 tablet_service.cc:1467] Processing CreateTablet for tablet 605bbc120f734820951ecb54606dfde6 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240506 11:41:02.609650 19746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 605bbc120f734820951ecb54606dfde6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.611254 19745 tablet_service.cc:1467] Processing CreateTablet for tablet c9423c59183d4a12a563adfe94f68648 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240506 11:41:02.611613 19745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9423c59183d4a12a563adfe94f68648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.620841 19811 heartbeater.cc:503] Master 127.10.45.190:37625 was elected leader, sending a full tablet report...
I20240506 11:41:02.629654 20095 tablet_bootstrap.cc:654] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.630903 20095 log.cc:824] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: Log is configured to *not* fsync() on all Append() calls
I20240506 11:41:02.634167 20094 tablet_bootstrap.cc:492] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963: Bootstrap starting.
I20240506 11:41:02.635947 20006 tablet_service.cc:1467] Processing CreateTablet for tablet 481042567046471aac1926a16d3a1265 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240506 11:41:02.637322 20006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481042567046471aac1926a16d3a1265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.637645 20094 tablet_bootstrap.cc:654] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.639041 20094 log.cc:824] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963: Log is configured to *not* fsync() on all Append() calls
I20240506 11:41:02.639266 19997 tablet_service.cc:1467] Processing CreateTablet for tablet 90737f5ce5dd4a4d8511d35ac96a0bcd (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240506 11:41:02.639715 19997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90737f5ce5dd4a4d8511d35ac96a0bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.640714 19998 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1c54d70c2448d1b4a18a6bdcc91eb3 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240506 11:41:02.641170 19998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1c54d70c2448d1b4a18a6bdcc91eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.643385 19999 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc17ce9c9a84dffa7cfa63adbdfce3e (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240506 11:41:02.643798 19999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc17ce9c9a84dffa7cfa63adbdfce3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.647325 20000 tablet_service.cc:1467] Processing CreateTablet for tablet 605bbc120f734820951ecb54606dfde6 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240506 11:41:02.647748 20000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 605bbc120f734820951ecb54606dfde6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.648826 20094 tablet_bootstrap.cc:492] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963: No bootstrap required, opened a new log
I20240506 11:41:02.649194 20094 ts_tablet_manager.cc:1367] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20240506 11:41:02.651376 20001 tablet_service.cc:1467] Processing CreateTablet for tablet c9423c59183d4a12a563adfe94f68648 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240506 11:41:02.651789 20001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9423c59183d4a12a563adfe94f68648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.655313 20002 tablet_service.cc:1467] Processing CreateTablet for tablet f7cd7fd9a7a947fb87505997aef18eb4 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240506 11:41:02.655718 20002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7cd7fd9a7a947fb87505997aef18eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.656348 20098 tablet_bootstrap.cc:492] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e: Bootstrap starting.
I20240506 11:41:02.659945 20003 tablet_service.cc:1467] Processing CreateTablet for tablet dee074a7650241048c409bd298d57f43 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240506 11:41:02.660257 20098 tablet_bootstrap.cc:654] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.660389 20003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee074a7650241048c409bd298d57f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.660473 20094 raft_consensus.cc:348] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.661069 20094 raft_consensus.cc:729] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74db6787b9e0470aa3fdd05a9dc3b963, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.661480 20098 log.cc:824] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e: Log is configured to *not* fsync() on all Append() calls
I20240506 11:41:02.661732 20094 consensus_queue.cc:260] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.667762 20005 tablet_service.cc:1467] Processing CreateTablet for tablet 83b67259f0024c23a55fb1ad46d21fe6 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240506 11:41:02.668316 20005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b67259f0024c23a55fb1ad46d21fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.669113 19941 heartbeater.cc:503] Master 127.10.45.190:37625 was elected leader, sending a full tablet report...
I20240506 11:41:02.670483 20094 ts_tablet_manager.cc:1398] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
I20240506 11:41:02.671183 20094 tablet_bootstrap.cc:492] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 74db6787b9e0470aa3fdd05a9dc3b963: Bootstrap starting.
I20240506 11:41:02.673173 20094 tablet_bootstrap.cc:654] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 74db6787b9e0470aa3fdd05a9dc3b963: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.676383 20095 tablet_bootstrap.cc:492] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: No bootstrap required, opened a new log
I20240506 11:41:02.676697 20095 ts_tablet_manager.cc:1367] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: Time spent bootstrapping tablet: real 0.083s user 0.020s sys 0.023s
I20240506 11:41:02.678467 20004 tablet_service.cc:1467] Processing CreateTablet for tablet 76cd0c3599094204a8bbfe14e296a750 (DEFAULT_TABLE table=test-workload [id=424b211b05b646138bff197aff5313a9]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240506 11:41:02.679782 20004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76cd0c3599094204a8bbfe14e296a750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 11:41:02.681288 20094 tablet_bootstrap.cc:492] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 74db6787b9e0470aa3fdd05a9dc3b963: No bootstrap required, opened a new log
I20240506 11:41:02.681488 20094 ts_tablet_manager.cc:1367] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20240506 11:41:02.682410 20094 raft_consensus.cc:348] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.682613 20094 raft_consensus.cc:729] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74db6787b9e0470aa3fdd05a9dc3b963, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.682792 20094 consensus_queue.cc:260] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 74db6787b9e0470aa3fdd05a9dc3b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.683712 20094 ts_tablet_manager.cc:1398] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 11:41:02.683948 20094 tablet_bootstrap.cc:492] T c9423c59183d4a12a563adfe94f68648 P 74db6787b9e0470aa3fdd05a9dc3b963: Bootstrap starting.
I20240506 11:41:02.685307 20094 tablet_bootstrap.cc:654] T c9423c59183d4a12a563adfe94f68648 P 74db6787b9e0470aa3fdd05a9dc3b963: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.685554 20095 raft_consensus.cc:348] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.685961 20095 raft_consensus.cc:729] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.686455 20095 consensus_queue.cc:260] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.689457 20094 tablet_bootstrap.cc:492] T c9423c59183d4a12a563adfe94f68648 P 74db6787b9e0470aa3fdd05a9dc3b963: No bootstrap required, opened a new log
I20240506 11:41:02.689605 20094 ts_tablet_manager.cc:1367] T c9423c59183d4a12a563adfe94f68648 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240506 11:41:02.690431 20094 raft_consensus.cc:348] T c9423c59183d4a12a563adfe94f68648 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.690709 20094 raft_consensus.cc:729] T c9423c59183d4a12a563adfe94f68648 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74db6787b9e0470aa3fdd05a9dc3b963, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.690927 20094 consensus_queue.cc:260] T c9423c59183d4a12a563adfe94f68648 P 74db6787b9e0470aa3fdd05a9dc3b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.692624 20094 ts_tablet_manager.cc:1398] T c9423c59183d4a12a563adfe94f68648 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 11:41:02.692947 20094 tablet_bootstrap.cc:492] T 605bbc120f734820951ecb54606dfde6 P 74db6787b9e0470aa3fdd05a9dc3b963: Bootstrap starting.
I20240506 11:41:02.693182 20098 tablet_bootstrap.cc:492] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e: No bootstrap required, opened a new log
I20240506 11:41:02.693480 20098 ts_tablet_manager.cc:1367] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.000s
I20240506 11:41:02.694842 20094 tablet_bootstrap.cc:654] T 605bbc120f734820951ecb54606dfde6 P 74db6787b9e0470aa3fdd05a9dc3b963: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.697062 20095 ts_tablet_manager.cc:1398] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: Time spent starting tablet: real 0.020s user 0.009s sys 0.010s
I20240506 11:41:02.697520 20095 tablet_bootstrap.cc:492] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: Bootstrap starting.
I20240506 11:41:02.699494 20095 tablet_bootstrap.cc:654] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.703027 20094 tablet_bootstrap.cc:492] T 605bbc120f734820951ecb54606dfde6 P 74db6787b9e0470aa3fdd05a9dc3b963: No bootstrap required, opened a new log
I20240506 11:41:02.703176 20094 ts_tablet_manager.cc:1367] T 605bbc120f734820951ecb54606dfde6 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240506 11:41:02.703306 20095 tablet_bootstrap.cc:492] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: No bootstrap required, opened a new log
I20240506 11:41:02.703462 20095 ts_tablet_manager.cc:1367] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.001s
I20240506 11:41:02.703769 20094 raft_consensus.cc:348] T 605bbc120f734820951ecb54606dfde6 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.703933 20094 raft_consensus.cc:729] T 605bbc120f734820951ecb54606dfde6 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74db6787b9e0470aa3fdd05a9dc3b963, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.704100 20094 consensus_queue.cc:260] T 605bbc120f734820951ecb54606dfde6 P 74db6787b9e0470aa3fdd05a9dc3b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.704313 20095 raft_consensus.cc:348] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.704517 20095 raft_consensus.cc:729] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.704576 20094 ts_tablet_manager.cc:1398] T 605bbc120f734820951ecb54606dfde6 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:02.704443 20098 raft_consensus.cc:348] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } } peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } }
I20240506 11:41:02.704668 20095 consensus_queue.cc:260] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.704804 20094 tablet_bootstrap.cc:492] T f7cd7fd9a7a947fb87505997aef18eb4 P 74db6787b9e0470aa3fdd05a9dc3b963: Bootstrap starting.
I20240506 11:41:02.705113 20095 ts_tablet_manager.cc:1398] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240506 11:41:02.705363 20095 tablet_bootstrap.cc:492] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: Bootstrap starting.
I20240506 11:41:02.706223 20094 tablet_bootstrap.cc:654] T f7cd7fd9a7a947fb87505997aef18eb4 P 74db6787b9e0470aa3fdd05a9dc3b963: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.704830 20098 raft_consensus.cc:729] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5414c092bba046b69d1cb7a07bc7e48e, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.706826 20095 tablet_bootstrap.cc:654] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.707213 20098 consensus_queue.cc:260] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } } peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } }
I20240506 11:41:02.728135 20094 tablet_bootstrap.cc:492] T f7cd7fd9a7a947fb87505997aef18eb4 P 74db6787b9e0470aa3fdd05a9dc3b963: No bootstrap required, opened a new log
I20240506 11:41:02.728320 20094 ts_tablet_manager.cc:1367] T f7cd7fd9a7a947fb87505997aef18eb4 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.016s
I20240506 11:41:02.728308 20071 heartbeater.cc:503] Master 127.10.45.190:37625 was elected leader, sending a full tablet report...
I20240506 11:41:02.728963 20094 raft_consensus.cc:348] T f7cd7fd9a7a947fb87505997aef18eb4 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.729132 20094 raft_consensus.cc:729] T f7cd7fd9a7a947fb87505997aef18eb4 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74db6787b9e0470aa3fdd05a9dc3b963, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.729301 20095 tablet_bootstrap.cc:492] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: No bootstrap required, opened a new log
I20240506 11:41:02.729292 20094 consensus_queue.cc:260] T f7cd7fd9a7a947fb87505997aef18eb4 P 74db6787b9e0470aa3fdd05a9dc3b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.729449 20095 ts_tablet_manager.cc:1367] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.018s
I20240506 11:41:02.729717 20094 ts_tablet_manager.cc:1398] T f7cd7fd9a7a947fb87505997aef18eb4 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:02.729947 20094 tablet_bootstrap.cc:492] T 76cd0c3599094204a8bbfe14e296a750 P 74db6787b9e0470aa3fdd05a9dc3b963: Bootstrap starting.
I20240506 11:41:02.730075 20095 raft_consensus.cc:348] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } } peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } }
I20240506 11:41:02.730278 20095 raft_consensus.cc:729] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.730458 20095 consensus_queue.cc:260] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } } peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } }
I20240506 11:41:02.731066 20095 ts_tablet_manager.cc:1398] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 11:41:02.731385 20094 tablet_bootstrap.cc:654] T 76cd0c3599094204a8bbfe14e296a750 P 74db6787b9e0470aa3fdd05a9dc3b963: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.731937 20095 tablet_bootstrap.cc:492] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: Bootstrap starting.
I20240506 11:41:02.732409 20098 ts_tablet_manager.cc:1398] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent starting tablet: real 0.039s user 0.021s sys 0.009s
I20240506 11:41:02.732750 20098 tablet_bootstrap.cc:492] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 5414c092bba046b69d1cb7a07bc7e48e: Bootstrap starting.
I20240506 11:41:02.733278 20095 tablet_bootstrap.cc:654] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.734436 20098 tablet_bootstrap.cc:654] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 5414c092bba046b69d1cb7a07bc7e48e: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.741428 20095 tablet_bootstrap.cc:492] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: No bootstrap required, opened a new log
I20240506 11:41:02.741613 20095 ts_tablet_manager.cc:1367] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240506 11:41:02.742396 20095 raft_consensus.cc:348] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.742604 20098 tablet_bootstrap.cc:492] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 5414c092bba046b69d1cb7a07bc7e48e: No bootstrap required, opened a new log
I20240506 11:41:02.742605 20095 raft_consensus.cc:729] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.742810 20098 ts_tablet_manager.cc:1367] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 5414c092bba046b69d1cb7a07bc7e48e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240506 11:41:02.742794 20095 consensus_queue.cc:260] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.743388 20095 ts_tablet_manager.cc:1398] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 11:41:02.743651 20095 tablet_bootstrap.cc:492] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: Bootstrap starting.
I20240506 11:41:02.745101 20095 tablet_bootstrap.cc:654] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.745434 20098 raft_consensus.cc:348] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.745652 20098 raft_consensus.cc:729] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5414c092bba046b69d1cb7a07bc7e48e, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.745820 20098 consensus_queue.cc:260] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 5414c092bba046b69d1cb7a07bc7e48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.746462 20098 ts_tablet_manager.cc:1398] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 5414c092bba046b69d1cb7a07bc7e48e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240506 11:41:02.746765 20098 tablet_bootstrap.cc:492] T dee074a7650241048c409bd298d57f43 P 5414c092bba046b69d1cb7a07bc7e48e: Bootstrap starting.
I20240506 11:41:02.748751 20098 tablet_bootstrap.cc:654] T dee074a7650241048c409bd298d57f43 P 5414c092bba046b69d1cb7a07bc7e48e: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.751184 20098 tablet_bootstrap.cc:492] T dee074a7650241048c409bd298d57f43 P 5414c092bba046b69d1cb7a07bc7e48e: No bootstrap required, opened a new log
I20240506 11:41:02.751369 20098 ts_tablet_manager.cc:1367] T dee074a7650241048c409bd298d57f43 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240506 11:41:02.752207 20098 raft_consensus.cc:348] T dee074a7650241048c409bd298d57f43 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.752492 20098 raft_consensus.cc:729] T dee074a7650241048c409bd298d57f43 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5414c092bba046b69d1cb7a07bc7e48e, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.752727 20098 consensus_queue.cc:260] T dee074a7650241048c409bd298d57f43 P 5414c092bba046b69d1cb7a07bc7e48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.753499 20098 ts_tablet_manager.cc:1398] T dee074a7650241048c409bd298d57f43 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 11:41:02.754729 20098 tablet_bootstrap.cc:492] T 83b67259f0024c23a55fb1ad46d21fe6 P 5414c092bba046b69d1cb7a07bc7e48e: Bootstrap starting.
I20240506 11:41:02.755996 20095 tablet_bootstrap.cc:492] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: No bootstrap required, opened a new log
I20240506 11:41:02.756150 20095 ts_tablet_manager.cc:1367] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240506 11:41:02.756678 20098 tablet_bootstrap.cc:654] T 83b67259f0024c23a55fb1ad46d21fe6 P 5414c092bba046b69d1cb7a07bc7e48e: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.756839 20095 raft_consensus.cc:348] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.757066 20095 raft_consensus.cc:729] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.757251 20095 consensus_queue.cc:260] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.757726 20095 ts_tablet_manager.cc:1398] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:02.757969 20095 tablet_bootstrap.cc:492] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: Bootstrap starting.
I20240506 11:41:02.759063 20094 tablet_bootstrap.cc:492] T 76cd0c3599094204a8bbfe14e296a750 P 74db6787b9e0470aa3fdd05a9dc3b963: No bootstrap required, opened a new log
I20240506 11:41:02.759215 20094 ts_tablet_manager.cc:1367] T 76cd0c3599094204a8bbfe14e296a750 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.009s
I20240506 11:41:02.759546 20095 tablet_bootstrap.cc:654] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.759907 20094 raft_consensus.cc:348] T 76cd0c3599094204a8bbfe14e296a750 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.760178 20094 raft_consensus.cc:729] T 76cd0c3599094204a8bbfe14e296a750 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74db6787b9e0470aa3fdd05a9dc3b963, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.760370 20094 consensus_queue.cc:260] T 76cd0c3599094204a8bbfe14e296a750 P 74db6787b9e0470aa3fdd05a9dc3b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.760833 20094 ts_tablet_manager.cc:1398] T 76cd0c3599094204a8bbfe14e296a750 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:02.761080 20094 tablet_bootstrap.cc:492] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963: Bootstrap starting.
I20240506 11:41:02.762704 20094 tablet_bootstrap.cc:654] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.765269 20095 tablet_bootstrap.cc:492] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: No bootstrap required, opened a new log
I20240506 11:41:02.765419 20095 ts_tablet_manager.cc:1367] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240506 11:41:02.766206 20098 tablet_bootstrap.cc:492] T 83b67259f0024c23a55fb1ad46d21fe6 P 5414c092bba046b69d1cb7a07bc7e48e: No bootstrap required, opened a new log
I20240506 11:41:02.766219 20095 raft_consensus.cc:348] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.766423 20095 raft_consensus.cc:729] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.766505 20098 ts_tablet_manager.cc:1367] T 83b67259f0024c23a55fb1ad46d21fe6 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240506 11:41:02.766542 20095 consensus_queue.cc:260] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.766738 20094 tablet_bootstrap.cc:492] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963: No bootstrap required, opened a new log
I20240506 11:41:02.766899 20094 ts_tablet_manager.cc:1367] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240506 11:41:02.767014 20095 ts_tablet_manager.cc:1398] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240506 11:41:02.767258 20095 tablet_bootstrap.cc:492] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: Bootstrap starting.
I20240506 11:41:02.767568 20094 raft_consensus.cc:348] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.767527 20098 raft_consensus.cc:348] T 83b67259f0024c23a55fb1ad46d21fe6 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.767755 20094 raft_consensus.cc:729] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74db6787b9e0470aa3fdd05a9dc3b963, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.767872 20098 raft_consensus.cc:729] T 83b67259f0024c23a55fb1ad46d21fe6 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5414c092bba046b69d1cb7a07bc7e48e, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.767952 20094 consensus_queue.cc:260] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.768074 20098 consensus_queue.cc:260] T 83b67259f0024c23a55fb1ad46d21fe6 P 5414c092bba046b69d1cb7a07bc7e48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.768667 20095 tablet_bootstrap.cc:654] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.768659 20098 ts_tablet_manager.cc:1398] T 83b67259f0024c23a55fb1ad46d21fe6 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 11:41:02.769001 20098 tablet_bootstrap.cc:492] T c9423c59183d4a12a563adfe94f68648 P 5414c092bba046b69d1cb7a07bc7e48e: Bootstrap starting.
I20240506 11:41:02.769055 20094 ts_tablet_manager.cc:1398] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240506 11:41:02.769294 20094 tablet_bootstrap.cc:492] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 74db6787b9e0470aa3fdd05a9dc3b963: Bootstrap starting.
I20240506 11:41:02.770593 20094 tablet_bootstrap.cc:654] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 74db6787b9e0470aa3fdd05a9dc3b963: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.770722 20098 tablet_bootstrap.cc:654] T c9423c59183d4a12a563adfe94f68648 P 5414c092bba046b69d1cb7a07bc7e48e: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.778975 20094 tablet_bootstrap.cc:492] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 74db6787b9e0470aa3fdd05a9dc3b963: No bootstrap required, opened a new log
I20240506 11:41:02.779141 20094 ts_tablet_manager.cc:1367] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240506 11:41:02.779898 20094 raft_consensus.cc:348] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.780154 20094 raft_consensus.cc:729] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74db6787b9e0470aa3fdd05a9dc3b963, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.780376 20094 consensus_queue.cc:260] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 74db6787b9e0470aa3fdd05a9dc3b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.781116 20094 ts_tablet_manager.cc:1398] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240506 11:41:02.781416 20094 tablet_bootstrap.cc:492] T 83b67259f0024c23a55fb1ad46d21fe6 P 74db6787b9e0470aa3fdd05a9dc3b963: Bootstrap starting.
I20240506 11:41:02.783331 20094 tablet_bootstrap.cc:654] T 83b67259f0024c23a55fb1ad46d21fe6 P 74db6787b9e0470aa3fdd05a9dc3b963: Neither blocks nor log segments found. Creating new log.
W20240506 11:41:02.786055 19942 tablet.cc:2369] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:41:02.787781 20095 tablet_bootstrap.cc:492] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: No bootstrap required, opened a new log
I20240506 11:41:02.787992 20095 ts_tablet_manager.cc:1367] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20240506 11:41:02.788817 20095 raft_consensus.cc:348] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.788983 20095 raft_consensus.cc:729] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.789505 20095 consensus_queue.cc:260] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.789659 20098 tablet_bootstrap.cc:492] T c9423c59183d4a12a563adfe94f68648 P 5414c092bba046b69d1cb7a07bc7e48e: No bootstrap required, opened a new log
I20240506 11:41:02.789894 20098 ts_tablet_manager.cc:1367] T c9423c59183d4a12a563adfe94f68648 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20240506 11:41:02.790658 20095 ts_tablet_manager.cc:1398] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240506 11:41:02.790657 20098 raft_consensus.cc:348] T c9423c59183d4a12a563adfe94f68648 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.790927 20095 tablet_bootstrap.cc:492] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: Bootstrap starting.
I20240506 11:41:02.791049 20098 raft_consensus.cc:729] T c9423c59183d4a12a563adfe94f68648 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5414c092bba046b69d1cb7a07bc7e48e, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.791275 20098 consensus_queue.cc:260] T c9423c59183d4a12a563adfe94f68648 P 5414c092bba046b69d1cb7a07bc7e48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.792352 20095 tablet_bootstrap.cc:654] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.792551 20098 ts_tablet_manager.cc:1398] T c9423c59183d4a12a563adfe94f68648 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240506 11:41:02.793243 20098 tablet_bootstrap.cc:492] T 605bbc120f734820951ecb54606dfde6 P 5414c092bba046b69d1cb7a07bc7e48e: Bootstrap starting.
I20240506 11:41:02.795336 20098 tablet_bootstrap.cc:654] T 605bbc120f734820951ecb54606dfde6 P 5414c092bba046b69d1cb7a07bc7e48e: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.810452 20094 tablet_bootstrap.cc:492] T 83b67259f0024c23a55fb1ad46d21fe6 P 74db6787b9e0470aa3fdd05a9dc3b963: No bootstrap required, opened a new log
I20240506 11:41:02.810601 20094 ts_tablet_manager.cc:1367] T 83b67259f0024c23a55fb1ad46d21fe6 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.012s
I20240506 11:41:02.810937 20095 tablet_bootstrap.cc:492] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: No bootstrap required, opened a new log
I20240506 11:41:02.811098 20095 ts_tablet_manager.cc:1367] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.012s
I20240506 11:41:02.811371 20098 tablet_bootstrap.cc:492] T 605bbc120f734820951ecb54606dfde6 P 5414c092bba046b69d1cb7a07bc7e48e: No bootstrap required, opened a new log
I20240506 11:41:02.811288 20094 raft_consensus.cc:348] T 83b67259f0024c23a55fb1ad46d21fe6 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.811491 20094 raft_consensus.cc:729] T 83b67259f0024c23a55fb1ad46d21fe6 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74db6787b9e0470aa3fdd05a9dc3b963, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.811601 20098 ts_tablet_manager.cc:1367] T 605bbc120f734820951ecb54606dfde6 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.006s
I20240506 11:41:02.811683 20094 consensus_queue.cc:260] T 83b67259f0024c23a55fb1ad46d21fe6 P 74db6787b9e0470aa3fdd05a9dc3b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.811869 20095 raft_consensus.cc:348] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.812076 20095 raft_consensus.cc:729] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.812259 20094 ts_tablet_manager.cc:1398] T 83b67259f0024c23a55fb1ad46d21fe6 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 11:41:02.812274 20095 consensus_queue.cc:260] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.812562 20094 tablet_bootstrap.cc:492] T 481042567046471aac1926a16d3a1265 P 74db6787b9e0470aa3fdd05a9dc3b963: Bootstrap starting.
I20240506 11:41:02.812502 20098 raft_consensus.cc:348] T 605bbc120f734820951ecb54606dfde6 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.812778 20098 raft_consensus.cc:729] T 605bbc120f734820951ecb54606dfde6 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5414c092bba046b69d1cb7a07bc7e48e, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.812819 20095 ts_tablet_manager.cc:1398] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240506 11:41:02.812974 20098 consensus_queue.cc:260] T 605bbc120f734820951ecb54606dfde6 P 5414c092bba046b69d1cb7a07bc7e48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.813534 20095 tablet_bootstrap.cc:492] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: Bootstrap starting.
I20240506 11:41:02.814003 20094 tablet_bootstrap.cc:654] T 481042567046471aac1926a16d3a1265 P 74db6787b9e0470aa3fdd05a9dc3b963: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.815030 20095 tablet_bootstrap.cc:654] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.815223 20098 ts_tablet_manager.cc:1398] T 605bbc120f734820951ecb54606dfde6 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240506 11:41:02.815507 20098 tablet_bootstrap.cc:492] T 76cd0c3599094204a8bbfe14e296a750 P 5414c092bba046b69d1cb7a07bc7e48e: Bootstrap starting.
I20240506 11:41:02.816921 20098 tablet_bootstrap.cc:654] T 76cd0c3599094204a8bbfe14e296a750 P 5414c092bba046b69d1cb7a07bc7e48e: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.825228 20098 tablet_bootstrap.cc:492] T 76cd0c3599094204a8bbfe14e296a750 P 5414c092bba046b69d1cb7a07bc7e48e: No bootstrap required, opened a new log
I20240506 11:41:02.825454 20098 ts_tablet_manager.cc:1367] T 76cd0c3599094204a8bbfe14e296a750 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240506 11:41:02.826126 20098 raft_consensus.cc:348] T 76cd0c3599094204a8bbfe14e296a750 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.826417 20098 raft_consensus.cc:729] T 76cd0c3599094204a8bbfe14e296a750 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5414c092bba046b69d1cb7a07bc7e48e, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.826645 20098 consensus_queue.cc:260] T 76cd0c3599094204a8bbfe14e296a750 P 5414c092bba046b69d1cb7a07bc7e48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.827369 20094 tablet_bootstrap.cc:492] T 481042567046471aac1926a16d3a1265 P 74db6787b9e0470aa3fdd05a9dc3b963: No bootstrap required, opened a new log
I20240506 11:41:02.827368 20098 ts_tablet_manager.cc:1398] T 76cd0c3599094204a8bbfe14e296a750 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240506 11:41:02.827574 20094 ts_tablet_manager.cc:1367] T 481042567046471aac1926a16d3a1265 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.012s
I20240506 11:41:02.827620 20098 tablet_bootstrap.cc:492] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e: Bootstrap starting.
I20240506 11:41:02.827780 20095 tablet_bootstrap.cc:492] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: No bootstrap required, opened a new log
I20240506 11:41:02.827944 20095 ts_tablet_manager.cc:1367] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20240506 11:41:02.828543 20094 raft_consensus.cc:348] T 481042567046471aac1926a16d3a1265 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } } peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } }
I20240506 11:41:02.828644 20095 raft_consensus.cc:348] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.828799 20094 raft_consensus.cc:729] T 481042567046471aac1926a16d3a1265 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74db6787b9e0470aa3fdd05a9dc3b963, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.828874 20095 raft_consensus.cc:729] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.829051 20095 consensus_queue.cc:260] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.829085 20094 consensus_queue.cc:260] T 481042567046471aac1926a16d3a1265 P 74db6787b9e0470aa3fdd05a9dc3b963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } } peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } }
I20240506 11:41:02.829464 20098 tablet_bootstrap.cc:654] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.830394 20094 ts_tablet_manager.cc:1398] T 481042567046471aac1926a16d3a1265 P 74db6787b9e0470aa3fdd05a9dc3b963: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240506 11:41:02.830888 20095 ts_tablet_manager.cc:1398] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240506 11:41:02.831192 20095 tablet_bootstrap.cc:492] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: Bootstrap starting.
I20240506 11:41:02.832691 20095 tablet_bootstrap.cc:654] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.834849 20098 tablet_bootstrap.cc:492] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e: No bootstrap required, opened a new log
I20240506 11:41:02.834863 20095 tablet_bootstrap.cc:492] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: No bootstrap required, opened a new log
I20240506 11:41:02.835032 20098 ts_tablet_manager.cc:1367] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240506 11:41:02.835032 20095 ts_tablet_manager.cc:1367] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240506 11:41:02.835711 20098 raft_consensus.cc:348] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.835711 20095 raft_consensus.cc:348] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.835937 20098 raft_consensus.cc:729] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5414c092bba046b69d1cb7a07bc7e48e, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.835935 20095 raft_consensus.cc:729] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.836135 20098 consensus_queue.cc:260] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.836138 20095 consensus_queue.cc:260] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.836625 20098 ts_tablet_manager.cc:1398] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:02.836624 20095 ts_tablet_manager.cc:1398] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:02.836877 20098 tablet_bootstrap.cc:492] T f7cd7fd9a7a947fb87505997aef18eb4 P 5414c092bba046b69d1cb7a07bc7e48e: Bootstrap starting.
I20240506 11:41:02.837918 20098 tablet_bootstrap.cc:654] T f7cd7fd9a7a947fb87505997aef18eb4 P 5414c092bba046b69d1cb7a07bc7e48e: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.839447 20098 tablet_bootstrap.cc:492] T f7cd7fd9a7a947fb87505997aef18eb4 P 5414c092bba046b69d1cb7a07bc7e48e: No bootstrap required, opened a new log
I20240506 11:41:02.839592 20098 ts_tablet_manager.cc:1367] T f7cd7fd9a7a947fb87505997aef18eb4 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240506 11:41:02.840128 20098 raft_consensus.cc:348] T f7cd7fd9a7a947fb87505997aef18eb4 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.840296 20098 raft_consensus.cc:729] T f7cd7fd9a7a947fb87505997aef18eb4 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5414c092bba046b69d1cb7a07bc7e48e, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.840456 20098 consensus_queue.cc:260] T f7cd7fd9a7a947fb87505997aef18eb4 P 5414c092bba046b69d1cb7a07bc7e48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.840884 20098 ts_tablet_manager.cc:1398] T f7cd7fd9a7a947fb87505997aef18eb4 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240506 11:41:02.841105 20098 tablet_bootstrap.cc:492] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 5414c092bba046b69d1cb7a07bc7e48e: Bootstrap starting.
I20240506 11:41:02.842139 20098 tablet_bootstrap.cc:654] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 5414c092bba046b69d1cb7a07bc7e48e: Neither blocks nor log segments found. Creating new log.
I20240506 11:41:02.843817 20098 tablet_bootstrap.cc:492] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 5414c092bba046b69d1cb7a07bc7e48e: No bootstrap required, opened a new log
I20240506 11:41:02.843955 20098 ts_tablet_manager.cc:1367] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240506 11:41:02.844539 20098 raft_consensus.cc:348] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.844734 20098 raft_consensus.cc:729] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5414c092bba046b69d1cb7a07bc7e48e, State: Initialized, Role: FOLLOWER
I20240506 11:41:02.844921 20098 consensus_queue.cc:260] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 5414c092bba046b69d1cb7a07bc7e48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.845355 20098 ts_tablet_manager.cc:1398] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 5414c092bba046b69d1cb7a07bc7e48e: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
W20240506 11:41:02.851384 19812 tablet.cc:2369] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:41:02.878474 19766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a1c54d70c2448d1b4a18a6bdcc91eb3"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:02.878780 19766 raft_consensus.cc:482] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:02.879011 19766 raft_consensus.cc:3037] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.881004 19766 raft_consensus.cc:504] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.882603 19766 leader_election.cc:288] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Requested vote from peers 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169), 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397)
I20240506 11:41:02.890367 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1c54d70c2448d1b4a18a6bdcc91eb3" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
I20240506 11:41:02.890758 19896 raft_consensus.cc:3037] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.890682 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1c54d70c2448d1b4a18a6bdcc91eb3" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5414c092bba046b69d1cb7a07bc7e48e"
I20240506 11:41:02.891045 20026 raft_consensus.cc:3037] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.892750 19896 raft_consensus.cc:2454] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.893013 20026 raft_consensus.cc:2454] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 5414c092bba046b69d1cb7a07bc7e48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.893316 19699 leader_election.cc:302] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74db6787b9e0470aa3fdd05a9dc3b963, 83434b0d0851482f95a9eaebcefb794d; no voters:
I20240506 11:41:02.893754 20103 raft_consensus.cc:2781] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:02.894194 20103 raft_consensus.cc:686] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Leader. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:02.894729 20103 consensus_queue.cc:237] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.899570 19625 catalog_manager.cc:5484] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83434b0d0851482f95a9eaebcefb794d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "83434b0d0851482f95a9eaebcefb794d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } health_report { overall_health: UNKNOWN } } }
I20240506 11:41:02.907054 19766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f7cd7fd9a7a947fb87505997aef18eb4"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:02.907254 19766 raft_consensus.cc:482] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:02.907351 19766 raft_consensus.cc:3037] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.909190 19766 raft_consensus.cc:504] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.909732 19766 leader_election.cc:288] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Requested vote from peers 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169), 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397)
I20240506 11:41:02.910144 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cd7fd9a7a947fb87505997aef18eb4" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
I20240506 11:41:02.910207 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cd7fd9a7a947fb87505997aef18eb4" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5414c092bba046b69d1cb7a07bc7e48e"
I20240506 11:41:02.910339 19896 raft_consensus.cc:3037] T f7cd7fd9a7a947fb87505997aef18eb4 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.910439 20026 raft_consensus.cc:3037] T f7cd7fd9a7a947fb87505997aef18eb4 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.912407 19896 raft_consensus.cc:2454] T f7cd7fd9a7a947fb87505997aef18eb4 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.912442 20026 raft_consensus.cc:2454] T f7cd7fd9a7a947fb87505997aef18eb4 P 5414c092bba046b69d1cb7a07bc7e48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.912863 19701 leader_election.cc:302] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5414c092bba046b69d1cb7a07bc7e48e, 74db6787b9e0470aa3fdd05a9dc3b963, 83434b0d0851482f95a9eaebcefb794d; no voters:
I20240506 11:41:02.913432 20103 raft_consensus.cc:2781] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:02.913592 20103 raft_consensus.cc:686] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Leader. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:02.913805 20103 consensus_queue.cc:237] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.916265 19625 catalog_manager.cc:5484] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83434b0d0851482f95a9eaebcefb794d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "83434b0d0851482f95a9eaebcefb794d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } health_report { overall_health: UNKNOWN } } }
I20240506 11:41:02.924721 19766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1bc17ce9c9a84dffa7cfa63adbdfce3e"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:02.924907 19766 raft_consensus.cc:482] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:02.925001 19766 raft_consensus.cc:3037] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.926690 19766 raft_consensus.cc:504] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.927290 19766 leader_election.cc:288] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Requested vote from peers 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169), 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397)
I20240506 11:41:02.927806 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc17ce9c9a84dffa7cfa63adbdfce3e" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
I20240506 11:41:02.927836 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc17ce9c9a84dffa7cfa63adbdfce3e" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5414c092bba046b69d1cb7a07bc7e48e"
I20240506 11:41:02.927982 19896 raft_consensus.cc:3037] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.928025 20026 raft_consensus.cc:3037] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.930025 19896 raft_consensus.cc:2454] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 74db6787b9e0470aa3fdd05a9dc3b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.930158 20026 raft_consensus.cc:2454] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.930420 19699 leader_election.cc:302] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74db6787b9e0470aa3fdd05a9dc3b963, 83434b0d0851482f95a9eaebcefb794d; no voters:
I20240506 11:41:02.930732 20103 raft_consensus.cc:2781] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:02.930892 20103 raft_consensus.cc:686] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Leader. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:02.931186 20103 consensus_queue.cc:237] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.933720 19625 catalog_manager.cc:5484] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83434b0d0851482f95a9eaebcefb794d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "83434b0d0851482f95a9eaebcefb794d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } health_report { overall_health: UNKNOWN } } }
I20240506 11:41:02.941792 19766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "76cd0c3599094204a8bbfe14e296a750"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:02.941989 19766 raft_consensus.cc:482] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:02.942081 19766 raft_consensus.cc:3037] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.943833 19766 raft_consensus.cc:504] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.944346 19766 leader_election.cc:288] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Requested vote from peers 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169), 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397)
I20240506 11:41:02.944766 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cd0c3599094204a8bbfe14e296a750" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
I20240506 11:41:02.944864 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76cd0c3599094204a8bbfe14e296a750" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5414c092bba046b69d1cb7a07bc7e48e"
I20240506 11:41:02.944993 19896 raft_consensus.cc:3037] T 76cd0c3599094204a8bbfe14e296a750 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.945097 20026 raft_consensus.cc:3037] T 76cd0c3599094204a8bbfe14e296a750 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.947032 19896 raft_consensus.cc:2454] T 76cd0c3599094204a8bbfe14e296a750 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.947230 20026 raft_consensus.cc:2454] T 76cd0c3599094204a8bbfe14e296a750 P 5414c092bba046b69d1cb7a07bc7e48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.947415 19699 leader_election.cc:302] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74db6787b9e0470aa3fdd05a9dc3b963, 83434b0d0851482f95a9eaebcefb794d; no voters:
I20240506 11:41:02.947738 20103 raft_consensus.cc:2781] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:02.947918 20103 raft_consensus.cc:686] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Leader. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:02.948169 20103 consensus_queue.cc:237] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.950727 19625 catalog_manager.cc:5484] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83434b0d0851482f95a9eaebcefb794d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "83434b0d0851482f95a9eaebcefb794d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } health_report { overall_health: UNKNOWN } } }
I20240506 11:41:02.957841 19766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90737f5ce5dd4a4d8511d35ac96a0bcd"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:02.958011 19766 raft_consensus.cc:482] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:02.958117 19766 raft_consensus.cc:3037] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.959867 19766 raft_consensus.cc:504] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.960398 19766 leader_election.cc:288] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Requested vote from peers 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169), 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397)
I20240506 11:41:02.960790 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90737f5ce5dd4a4d8511d35ac96a0bcd" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
I20240506 11:41:02.960907 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90737f5ce5dd4a4d8511d35ac96a0bcd" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5414c092bba046b69d1cb7a07bc7e48e"
I20240506 11:41:02.960992 19896 raft_consensus.cc:3037] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.961094 20026 raft_consensus.cc:3037] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.963761 20026 raft_consensus.cc:2454] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 5414c092bba046b69d1cb7a07bc7e48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.963761 19896 raft_consensus.cc:2454] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 74db6787b9e0470aa3fdd05a9dc3b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.964141 19699 leader_election.cc:302] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74db6787b9e0470aa3fdd05a9dc3b963, 83434b0d0851482f95a9eaebcefb794d; no voters:
I20240506 11:41:02.964433 20103 raft_consensus.cc:2781] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:02.964579 20103 raft_consensus.cc:686] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Leader. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:02.964823 20103 consensus_queue.cc:237] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.967144 19625 catalog_manager.cc:5484] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83434b0d0851482f95a9eaebcefb794d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "83434b0d0851482f95a9eaebcefb794d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } health_report { overall_health: UNKNOWN } } }
W20240506 11:41:02.974706 20072 tablet.cc:2369] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 5414c092bba046b69d1cb7a07bc7e48e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 11:41:02.974891 19766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83b67259f0024c23a55fb1ad46d21fe6"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:02.975100 19766 raft_consensus.cc:482] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:02.975193 19766 raft_consensus.cc:3037] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.976812 19766 raft_consensus.cc:504] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.977285 19766 leader_election.cc:288] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Requested vote from peers 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169), 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397)
I20240506 11:41:02.977756 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b67259f0024c23a55fb1ad46d21fe6" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
I20240506 11:41:02.977780 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b67259f0024c23a55fb1ad46d21fe6" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5414c092bba046b69d1cb7a07bc7e48e"
I20240506 11:41:02.977982 20026 raft_consensus.cc:3037] T 83b67259f0024c23a55fb1ad46d21fe6 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.977981 19896 raft_consensus.cc:3037] T 83b67259f0024c23a55fb1ad46d21fe6 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.980623 19896 raft_consensus.cc:2454] T 83b67259f0024c23a55fb1ad46d21fe6 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.980623 20026 raft_consensus.cc:2454] T 83b67259f0024c23a55fb1ad46d21fe6 P 5414c092bba046b69d1cb7a07bc7e48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.981009 19699 leader_election.cc:302] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74db6787b9e0470aa3fdd05a9dc3b963, 83434b0d0851482f95a9eaebcefb794d; no voters:
I20240506 11:41:02.981245 20103 raft_consensus.cc:2781] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:02.981396 20103 raft_consensus.cc:686] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Leader. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:02.981626 20103 consensus_queue.cc:237] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:02.984197 19625 catalog_manager.cc:5484] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83434b0d0851482f95a9eaebcefb794d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "83434b0d0851482f95a9eaebcefb794d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } health_report { overall_health: UNKNOWN } } }
I20240506 11:41:02.991556 19766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "481042567046471aac1926a16d3a1265"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:02.991691 19766 raft_consensus.cc:482] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:02.991775 19766 raft_consensus.cc:3037] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.993348 19766 raft_consensus.cc:504] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } } peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } }
I20240506 11:41:02.993818 19766 leader_election.cc:288] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Requested vote from peers 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397), 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169)
I20240506 11:41:02.994225 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481042567046471aac1926a16d3a1265" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5414c092bba046b69d1cb7a07bc7e48e"
I20240506 11:41:02.994269 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481042567046471aac1926a16d3a1265" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
I20240506 11:41:02.994386 20026 raft_consensus.cc:3037] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.994469 19896 raft_consensus.cc:3037] T 481042567046471aac1926a16d3a1265 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:02.996273 20026 raft_consensus.cc:2454] T 481042567046471aac1926a16d3a1265 P 5414c092bba046b69d1cb7a07bc7e48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.996306 19896 raft_consensus.cc:2454] T 481042567046471aac1926a16d3a1265 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:02.996636 19701 leader_election.cc:302] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5414c092bba046b69d1cb7a07bc7e48e, 83434b0d0851482f95a9eaebcefb794d; no voters:
I20240506 11:41:02.996922 20103 raft_consensus.cc:2781] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:02.997076 20103 raft_consensus.cc:686] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Leader. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:02.997315 20103 consensus_queue.cc:237] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } } peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } }
I20240506 11:41:03.000065 19625 catalog_manager.cc:5484] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83434b0d0851482f95a9eaebcefb794d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "83434b0d0851482f95a9eaebcefb794d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } health_report { overall_health: UNKNOWN } } }
I20240506 11:41:03.006614 19766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9423c59183d4a12a563adfe94f68648"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.006780 19766 raft_consensus.cc:482] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:03.006871 19766 raft_consensus.cc:3037] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:03.008551 19766 raft_consensus.cc:504] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:03.009055 19766 leader_election.cc:288] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Requested vote from peers 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169), 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397)
I20240506 11:41:03.009414 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9423c59183d4a12a563adfe94f68648" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
I20240506 11:41:03.009466 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9423c59183d4a12a563adfe94f68648" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5414c092bba046b69d1cb7a07bc7e48e"
I20240506 11:41:03.009636 19896 raft_consensus.cc:3037] T c9423c59183d4a12a563adfe94f68648 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:03.009660 20026 raft_consensus.cc:3037] T c9423c59183d4a12a563adfe94f68648 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:03.012272 20026 raft_consensus.cc:2454] T c9423c59183d4a12a563adfe94f68648 P 5414c092bba046b69d1cb7a07bc7e48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:03.012272 19896 raft_consensus.cc:2454] T c9423c59183d4a12a563adfe94f68648 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:03.012643 19701 leader_election.cc:302] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5414c092bba046b69d1cb7a07bc7e48e, 83434b0d0851482f95a9eaebcefb794d; no voters:
I20240506 11:41:03.012872 20103 raft_consensus.cc:2781] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:03.013038 20103 raft_consensus.cc:686] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Leader. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:03.013262 20103 consensus_queue.cc:237] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:03.015630 19625 catalog_manager.cc:5484] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83434b0d0851482f95a9eaebcefb794d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "83434b0d0851482f95a9eaebcefb794d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } health_report { overall_health: UNKNOWN } } }
I20240506 11:41:03.024096 19766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "605bbc120f734820951ecb54606dfde6"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.024235 19766 raft_consensus.cc:482] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:03.024318 19766 raft_consensus.cc:3037] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:03.025892 19766 raft_consensus.cc:504] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:03.026356 19766 leader_election.cc:288] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Requested vote from peers 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169), 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397)
I20240506 11:41:03.026691 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605bbc120f734820951ecb54606dfde6" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
I20240506 11:41:03.026888 19896 raft_consensus.cc:3037] T 605bbc120f734820951ecb54606dfde6 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:03.026794 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "605bbc120f734820951ecb54606dfde6" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5414c092bba046b69d1cb7a07bc7e48e"
I20240506 11:41:03.027024 20026 raft_consensus.cc:3037] T 605bbc120f734820951ecb54606dfde6 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:03.028733 19896 raft_consensus.cc:2454] T 605bbc120f734820951ecb54606dfde6 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:03.028939 20026 raft_consensus.cc:2454] T 605bbc120f734820951ecb54606dfde6 P 5414c092bba046b69d1cb7a07bc7e48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:03.029081 19699 leader_election.cc:302] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74db6787b9e0470aa3fdd05a9dc3b963, 83434b0d0851482f95a9eaebcefb794d; no voters:
I20240506 11:41:03.029356 20103 raft_consensus.cc:2781] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:03.029486 20103 raft_consensus.cc:686] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Leader. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:03.029701 20103 consensus_queue.cc:237] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:03.031831 19625 catalog_manager.cc:5484] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83434b0d0851482f95a9eaebcefb794d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "83434b0d0851482f95a9eaebcefb794d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } health_report { overall_health: UNKNOWN } } }
I20240506 11:41:03.040472 19766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dee074a7650241048c409bd298d57f43"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.040639 19766 raft_consensus.cc:482] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 11:41:03.040726 19766 raft_consensus.cc:3037] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:03.042418 19766 raft_consensus.cc:504] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:03.042878 19766 leader_election.cc:288] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Requested vote from peers 74db6787b9e0470aa3fdd05a9dc3b963 (127.10.45.130:41169), 5414c092bba046b69d1cb7a07bc7e48e (127.10.45.131:44397)
I20240506 11:41:03.043453 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee074a7650241048c409bd298d57f43" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5414c092bba046b69d1cb7a07bc7e48e"
I20240506 11:41:03.043641 20026 raft_consensus.cc:3037] T dee074a7650241048c409bd298d57f43 P 5414c092bba046b69d1cb7a07bc7e48e [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:03.043707 19896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee074a7650241048c409bd298d57f43" candidate_uuid: "83434b0d0851482f95a9eaebcefb794d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74db6787b9e0470aa3fdd05a9dc3b963"
I20240506 11:41:03.043998 19896 raft_consensus.cc:3037] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 0 FOLLOWER]: Advancing to term 1
I20240506 11:41:03.045562 20026 raft_consensus.cc:2454] T dee074a7650241048c409bd298d57f43 P 5414c092bba046b69d1cb7a07bc7e48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:03.045887 19701 leader_election.cc:302] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5414c092bba046b69d1cb7a07bc7e48e, 83434b0d0851482f95a9eaebcefb794d; no voters:
I20240506 11:41:03.046104 20103 raft_consensus.cc:2781] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Leader election won for term 1
I20240506 11:41:03.046258 20103 raft_consensus.cc:686] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Leader. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:03.046483 20103 consensus_queue.cc:237] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:03.046696 19896 raft_consensus.cc:2454] T dee074a7650241048c409bd298d57f43 P 74db6787b9e0470aa3fdd05a9dc3b963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83434b0d0851482f95a9eaebcefb794d in term 1.
I20240506 11:41:03.048969 19625 catalog_manager.cc:5484] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83434b0d0851482f95a9eaebcefb794d (127.10.45.129). New cstate: current_term: 1 leader_uuid: "83434b0d0851482f95a9eaebcefb794d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } health_report { overall_health: UNKNOWN } } }
I20240506 11:41:03.058107 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a1c54d70c2448d1b4a18a6bdcc91eb3 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.058843 20146 tablet_replica.cc:330] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: stopping tablet replica
I20240506 11:41:03.059260 20146 raft_consensus.cc:2229] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Raft consensus shutting down.
I20240506 11:41:03.059764 20146 pending_rounds.cc:62] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: Trying to abort 1 pending ops.
I20240506 11:41:03.059906 20146 pending_rounds.cc:68] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024622235221663744 op_type: NO_OP noop_request { }
I20240506 11:41:03.060101 20146 raft_consensus.cc:2866] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 11:41:03.060217 20146 raft_consensus.cc:2258] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:03.061659 20146 ts_tablet_manager.cc:1875] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:03.066149 20146 ts_tablet_manager.cc:1888] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:03.066318 20146 log.cc:1196] T 9a1c54d70c2448d1b4a18a6bdcc91eb3 P 83434b0d0851482f95a9eaebcefb794d: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/9a1c54d70c2448d1b4a18a6bdcc91eb3
I20240506 11:41:03.074986 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd7fd9a7a947fb87505997aef18eb4 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.075265 20146 tablet_replica.cc:330] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: stopping tablet replica
I20240506 11:41:03.075477 20146 raft_consensus.cc:2229] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Raft consensus shutting down.
I20240506 11:41:03.075736 20146 pending_rounds.cc:62] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: Trying to abort 1 pending ops.
I20240506 11:41:03.075824 20146 pending_rounds.cc:68] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024622235296903168 op_type: NO_OP noop_request { }
I20240506 11:41:03.075940 20146 raft_consensus.cc:2866] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 11:41:03.076061 20146 raft_consensus.cc:2258] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:03.076771 20146 ts_tablet_manager.cc:1875] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:03.080765 20146 ts_tablet_manager.cc:1888] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:03.080885 20146 log.cc:1196] T f7cd7fd9a7a947fb87505997aef18eb4 P 83434b0d0851482f95a9eaebcefb794d: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7cd7fd9a7a947fb87505997aef18eb4
I20240506 11:41:03.089185 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc17ce9c9a84dffa7cfa63adbdfce3e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.089463 20146 tablet_replica.cc:330] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: stopping tablet replica
I20240506 11:41:03.089640 20146 raft_consensus.cc:2229] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Raft consensus shutting down.
I20240506 11:41:03.089900 20146 pending_rounds.cc:62] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: Trying to abort 1 pending ops.
I20240506 11:41:03.089972 20146 pending_rounds.cc:68] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024622235368128512 op_type: NO_OP noop_request { }
I20240506 11:41:03.090103 20146 raft_consensus.cc:2866] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 11:41:03.090198 20146 raft_consensus.cc:2258] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:03.090989 20146 ts_tablet_manager.cc:1875] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:03.094601 20146 ts_tablet_manager.cc:1888] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:03.094708 20146 log.cc:1196] T 1bc17ce9c9a84dffa7cfa63adbdfce3e P 83434b0d0851482f95a9eaebcefb794d: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/1bc17ce9c9a84dffa7cfa63adbdfce3e
I20240506 11:41:03.107901 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet 76cd0c3599094204a8bbfe14e296a750 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.108119 20146 tablet_replica.cc:330] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: stopping tablet replica
I20240506 11:41:03.108299 20146 raft_consensus.cc:2229] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Raft consensus shutting down.
I20240506 11:41:03.108505 20146 pending_rounds.cc:62] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: Trying to abort 1 pending ops.
I20240506 11:41:03.108577 20146 pending_rounds.cc:68] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024622235437756416 op_type: NO_OP noop_request { }
I20240506 11:41:03.108695 20146 raft_consensus.cc:2866] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 11:41:03.108801 20146 raft_consensus.cc:2258] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:03.109503 20146 ts_tablet_manager.cc:1875] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:03.113400 20146 ts_tablet_manager.cc:1888] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:03.113517 20146 log.cc:1196] T 76cd0c3599094204a8bbfe14e296a750 P 83434b0d0851482f95a9eaebcefb794d: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/76cd0c3599094204a8bbfe14e296a750
I20240506 11:41:03.121793 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet 90737f5ce5dd4a4d8511d35ac96a0bcd with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.122061 20146 tablet_replica.cc:330] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: stopping tablet replica
I20240506 11:41:03.122229 20146 raft_consensus.cc:2229] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Raft consensus shutting down.
I20240506 11:41:03.122467 20146 pending_rounds.cc:62] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: Trying to abort 1 pending ops.
I20240506 11:41:03.122520 20146 pending_rounds.cc:68] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024622235505664000 op_type: NO_OP noop_request { }
I20240506 11:41:03.122691 20146 raft_consensus.cc:2866] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240506 11:41:03.122794 20146 raft_consensus.cc:2258] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:03.123517 20146 ts_tablet_manager.cc:1875] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:03.127261 20146 ts_tablet_manager.cc:1888] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:03.127375 20146 log.cc:1196] T 90737f5ce5dd4a4d8511d35ac96a0bcd P 83434b0d0851482f95a9eaebcefb794d: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/90737f5ce5dd4a4d8511d35ac96a0bcd
I20240506 11:41:03.135350 19766 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83b67259f0024c23a55fb1ad46d21fe6"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.135530 19766 raft_consensus.cc:580] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Received request to step down
I20240506 11:41:03.135605 19766 raft_consensus.cc:3032] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:41:03.135649 19766 raft_consensus.cc:729] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:03.135862 19766 consensus_queue.cc:260] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:03.136118 19766 raft_consensus.cc:3037] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:41:03.143828 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b67259f0024c23a55fb1ad46d21fe6 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.144065 20146 tablet_replica.cc:330] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: stopping tablet replica
I20240506 11:41:03.144249 20146 raft_consensus.cc:2229] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:41:03.144387 20146 pending_rounds.cc:62] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: Trying to abort 1 pending ops.
I20240506 11:41:03.144446 20146 pending_rounds.cc:68] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024622235574853632 op_type: NO_OP noop_request { }
I20240506 11:41:03.144549 20146 raft_consensus.cc:2866] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240506 11:41:03.144654 20146 raft_consensus.cc:2258] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:03.145340 20146 ts_tablet_manager.cc:1875] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:03.148988 20146 ts_tablet_manager.cc:1888] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:03.149080 20146 log.cc:1196] T 83b67259f0024c23a55fb1ad46d21fe6 P 83434b0d0851482f95a9eaebcefb794d: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/83b67259f0024c23a55fb1ad46d21fe6
I20240506 11:41:03.156994 19766 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "481042567046471aac1926a16d3a1265"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.157145 19766 raft_consensus.cc:580] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Received request to step down
I20240506 11:41:03.157218 19766 raft_consensus.cc:3032] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:41:03.157265 19766 raft_consensus.cc:729] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:03.157429 19766 consensus_queue.cc:260] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } } peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } }
I20240506 11:41:03.157617 19766 raft_consensus.cc:3037] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:41:03.165303 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet 481042567046471aac1926a16d3a1265 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.165549 20146 tablet_replica.cc:330] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: stopping tablet replica
I20240506 11:41:03.165724 20146 raft_consensus.cc:2229] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:41:03.165866 20146 pending_rounds.cc:62] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: Trying to abort 1 pending ops.
I20240506 11:41:03.165937 20146 pending_rounds.cc:68] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024622235638788096 op_type: NO_OP noop_request { }
I20240506 11:41:03.166081 20146 raft_consensus.cc:2866] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240506 11:41:03.166175 20146 raft_consensus.cc:2258] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:03.166929 20146 ts_tablet_manager.cc:1875] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:03.170677 20146 ts_tablet_manager.cc:1888] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:03.170810 20146 log.cc:1196] T 481042567046471aac1926a16d3a1265 P 83434b0d0851482f95a9eaebcefb794d: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/481042567046471aac1926a16d3a1265
I20240506 11:41:03.178874 19766 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9423c59183d4a12a563adfe94f68648"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.179077 19766 raft_consensus.cc:580] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Received request to step down
I20240506 11:41:03.179172 19766 raft_consensus.cc:3032] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:41:03.179239 19766 raft_consensus.cc:729] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:03.179468 19766 consensus_queue.cc:260] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:03.179683 19766 raft_consensus.cc:3037] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:41:03.187391 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet c9423c59183d4a12a563adfe94f68648 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.187633 20146 tablet_replica.cc:330] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: stopping tablet replica
I20240506 11:41:03.187812 20146 raft_consensus.cc:2229] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:41:03.187947 20146 pending_rounds.cc:62] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: Trying to abort 1 pending ops.
I20240506 11:41:03.188016 20146 pending_rounds.cc:68] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024622235704168448 op_type: NO_OP noop_request { }
I20240506 11:41:03.188138 20146 raft_consensus.cc:2866] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240506 11:41:03.188248 20146 raft_consensus.cc:2258] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:03.188987 20146 ts_tablet_manager.cc:1875] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:03.192803 20146 ts_tablet_manager.cc:1888] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:03.192894 20146 log.cc:1196] T c9423c59183d4a12a563adfe94f68648 P 83434b0d0851482f95a9eaebcefb794d: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/c9423c59183d4a12a563adfe94f68648
I20240506 11:41:03.201284 19766 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "605bbc120f734820951ecb54606dfde6"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.201459 19766 raft_consensus.cc:580] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Received request to step down
I20240506 11:41:03.201560 19766 raft_consensus.cc:3032] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:41:03.201632 19766 raft_consensus.cc:729] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:03.201864 19766 consensus_queue.cc:260] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:03.202080 19766 raft_consensus.cc:3037] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:41:03.209817 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet 605bbc120f734820951ecb54606dfde6 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.210081 20146 tablet_replica.cc:330] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: stopping tablet replica
I20240506 11:41:03.210253 20146 raft_consensus.cc:2229] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:41:03.210431 20146 pending_rounds.cc:62] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: Trying to abort 1 pending ops.
I20240506 11:41:03.210486 20146 pending_rounds.cc:68] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024622235771424768 op_type: NO_OP noop_request { }
I20240506 11:41:03.210623 20146 raft_consensus.cc:2866] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240506 11:41:03.210716 20146 raft_consensus.cc:2258] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:03.211496 20146 ts_tablet_manager.cc:1875] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:03.215381 20146 ts_tablet_manager.cc:1888] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:03.215500 20146 log.cc:1196] T 605bbc120f734820951ecb54606dfde6 P 83434b0d0851482f95a9eaebcefb794d: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/605bbc120f734820951ecb54606dfde6
I20240506 11:41:03.223536 19766 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dee074a7650241048c409bd298d57f43"
dest_uuid: "83434b0d0851482f95a9eaebcefb794d"
from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.223701 19766 raft_consensus.cc:580] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Received request to step down
I20240506 11:41:03.223805 19766 raft_consensus.cc:3032] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Stepping down as leader of term 1
I20240506 11:41:03.223870 19766 raft_consensus.cc:729] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83434b0d0851482f95a9eaebcefb794d, State: Running, Role: LEADER
I20240506 11:41:03.224072 19766 consensus_queue.cc:260] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83434b0d0851482f95a9eaebcefb794d" member_type: VOTER last_known_addr { host: "127.10.45.129" port: 43585 } } peers { permanent_uuid: "74db6787b9e0470aa3fdd05a9dc3b963" member_type: VOTER last_known_addr { host: "127.10.45.130" port: 41169 } } peers { permanent_uuid: "5414c092bba046b69d1cb7a07bc7e48e" member_type: VOTER last_known_addr { host: "127.10.45.131" port: 44397 } }
I20240506 11:41:03.224292 19766 raft_consensus.cc:3037] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 1 FOLLOWER]: Advancing to term 2
I20240506 11:41:03.232280 19741 tablet_service.cc:1514] Processing DeleteTablet for tablet dee074a7650241048c409bd298d57f43 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49478
I20240506 11:41:03.232527 20146 tablet_replica.cc:330] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: stopping tablet replica
I20240506 11:41:03.232703 20146 raft_consensus.cc:2229] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240506 11:41:03.232834 20146 pending_rounds.cc:62] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: Trying to abort 1 pending ops.
I20240506 11:41:03.232916 20146 pending_rounds.cc:68] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024622235840753664 op_type: NO_OP noop_request { }
I20240506 11:41:03.233031 20146 raft_consensus.cc:2866] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240506 11:41:03.233139 20146 raft_consensus.cc:2258] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240506 11:41:03.233866 20146 ts_tablet_manager.cc:1875] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240506 11:41:03.237738 20146 ts_tablet_manager.cc:1888] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20240506 11:41:03.237830 20146 log.cc:1196] T dee074a7650241048c409bd298d57f43 P 83434b0d0851482f95a9eaebcefb794d: Deleting WAL directory at /tmp/dist-test-taskNkhkL1/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1714995474491875-10422-0/raft_consensus-itest-cluster/ts-0/wal/wals/dee074a7650241048c409bd298d57f43
I20240506 11:41:03.247238 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 19684
I20240506 11:41:03.257413 19806 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:41:03.366576 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 19684
I20240506 11:41:03.386206 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 19814
I20240506 11:41:03.391808 19936 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:41:03.483599 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 19814
I20240506 11:41:03.506628 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 19944
I20240506 11:41:03.512573 20066 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:41:03.603381 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 19944
I20240506 11:41:03.620496 10422 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu pid 19594
I20240506 11:41:03.621403 19655 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 11:41:03.720613 10422 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNkhkL1/build/asan/bin/kudu with pid 19594
2024-05-06T11:41:03Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1 (3102 ms)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest (4758 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 2 test suites ran. (189210 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
1 FAILED TEST
I20240506 11:41:03.765295 10422 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 53 messages since previous log ~15 seconds ago